In this case I needed to create a glance image that could be deployed to a openstack cluster … and that is where the fun stops.

First things first, if you can do a clean install (if you paid the extra £20 and actually received your dvd media that is!) do so, the upgrade process from Windows 7 took the best part of 2 days to complete.

Secondly to create your glance image you’re going to have to do the installation on the same type of hypervisor that you have openstack running upon, in this case I will be covering deployment of Windows 8 onto Linux KVM with virtio drivers.

The kludge

You can not start the instance using virtio for the hard disk, it simply puts itself into a never ending recovery mode, instead set the bus type to SATA or IDE.

Attach a second drive that uses virtio bus, why you may ask? Windows 8 will now boot and in turn have a device attached which it can not recognize.

Before booting you will also need to attach this iso as a cdrom, at the time of writing you can use the Win7 drivers for Windows 8. (iso version 0.1-30)

Square peg, round hole == Bigger hammer

I opted to first install all the drivers by opening up the virtual cdrom, navigating to the Win7 folder and: right click -> install on all the “Setup Information” files.

My “fun” did not end here however … because it would appear the attached virtio device was not formatted Windows8 decided to ignore it.

In this case the device manager needs to be launched to resolve the issue a laborious task in itelf.

1. Open desktop, and click the windows explorer tray icon.
2. Right click “Computer” and click properties.
3. Click “Device Manager”.
4. Expand the “Disk Drives” section, (if you did not install the drivers and reboot, you may be prompted to install the device, or it will show up as an unknown device instead of a disk drive)
5. Right click properties on the “RedHat VirtIO SCSI Device”
6. Click the volumes tab and click populate.
7. Close all windows leaving the Explorer window open.
8. Right click computer, select Manage.
9. Select disk management, partition and format the Virtio device as you would any other hard drive.
10. You should now have a new volume, this is running with the virtio drivers.
11. Shutdown windows.
12. Reconfigure the KVM instance, remove the VirtIO disk, change the primary disk bus to VirtIO
13. Start windows, and wait … and wait …
14. Once the start menu has booted you will begin to notice performance picks up after a while, I assume this is due to background tasks running.
15. Run any updates that may be outstanding and shutdown the instance. I would also advise configuring remove desktop
16. Convert to qcow2 (if you want), and import into glance as you would any other image.
17. Create or modify a security group if you have opted to allow Remote Desktop.

And boot the image as normal, ensuring that the selected “flavor” has enough disk space to start the instance.

As for meta data injection, for say account setup I have no idea at this time, please feel free to post in the comment or email me with methods for doing so.

Credits

this blog for noting the ‘dirty hack’ workaround in Windows 8 R2

and James P for having way more patience with windows than I will ever have.

unable to load certificate 140735207381436:error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1319: 140735207381436:error:0D07803A:asn1 encoding routines:ASN1_ITEM_EX_D2I:nested asn1 error:tasn_dec.c:381:Type=X509_CINF 140735207381436:error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:751:Field=cert_info, Type=X509 140735207381436:error:0906700D:PEM routines:PEM_ASN1_read_bio:ASN1 lib:pem_oth.c:83:

When trying to validate a certificate using openssl, this is because it is in the wrong format, whilst the certificate file visually appears to be in x.509 format, you will find it contains a far longer base64 string than x.509 certificats of the same bit length.

The format in this case is p7b (PCKS #7); to use the certificate witih apache you’re going to have to convert this.

openssl pkcs7 -print_certs -in certificate.p7b -out certificate.cer

Within the resulting .cer file you will file you x.509 certificate bundled with relevant CA certificates, break these out into your relevant .crt and ca.crt files and load as normal into apache.

After an initial read the setup process seemed very simple, and as it would turn out it was; I later moved onto some simple resillience testing of my 4 node p.o.c. cluster.

I’m still a little unsure on the circular topology I ended up using; but it appears absolutely fine so long as the following conditions are met.

1. At least one node is always available.
2. Nodes are recovered only if their peer is available to sync from. a. Requiring a startup order.

This is not such a bad thing, as if all nodes were to suddenly go down; I can’t think of a situation where you would want it all to recover “automagically” you would want to inspect to ensure data integrity and recover from a “known good” version of your data.

Openstack as an experimentation platform

Openstack i I’ve found perfect for rapid prototyping of hostinsg platform architectures, in none geek building virtual models of servers and services; ensuring sure they all go together properly before committing to the build plan.

The best part being the VM’s are “Throw away”, something goes inexplicably wrong with a vm prototype? assuming you used snapshots at each step it’s easy enough to roll back.

For reference I used Fedora 17 and the wiki reference setup of openstack for prototyping.

Note in this case you may be better off using OpenVZ; whilst openstack does not at the time of writing support this directly, the openstack DBaaS (Database as a Service) project Red Dwarf leverages OpenVZ to provide DBaaS, (Something I’d like to get auto handeling clusters via XtraDB clustering, given the time …).

XtraDB cluster p.o.c. platform

My platform consists of 4 nodes; although I am sured an odd number of nodes is preferable to reduce the risk of split-brain behaviour occuring.

Nginx is not exempt from security issues, and as with apache certain versions can vulnerable to a specific attack, as such the first line of defense is you hide your nginx version.

This can be done via:

1
2
3
4

server {
    server_tokens off;
    ...
}

This changes the put from

1

Server: nginx/1.0.12

To

1

Server: nginx

You could if you are so inclined change the server string in the c code itself

src/http/ngx_http_header_filter_module.c

...
static char ngx_http_server_string[] = "Server: my_modified_server" CRLF;
static char ngx_http_server_full_string[] = "Server: my_modified_server/release_version" CRLF;
...

To err is human …

Sometimes standard responses can be used for service fingerprinting as such error documents could still give away your running server version even if you were to edit the header code as per above, again this could be done by modifying the C code to only return “” for each error page, in which case you will need to edit

src/http/ngx_http_special_response.c

...
static char ngx_http_error_301_page[] = "";

I’m not going to list all of them you should get the idea from the exmaple above; however this is not really required, you can also swap out the default error pages with standard configuration.

1

error_page 404 = /path/to/custom/404.html;

A strong Front …

Nginx ofetn gets used to proxy other services, as such you could be revealing the backend technologies in use due to the backend server sending headers such as X-Powered-By.

This where in your proxy configure options you can have nginx intercept and remove the headers being sent by the backend.

1

proxy_hide_headers X-Powered-By;

For this you will require the following Packages:

1. libguestfs-tools
2. guestfish

Shutdown the instance

In order to grow the disk we must virsh shutdown the instance, this can be achieved using a simple virsh shutdown instance_name, try to avoid running a virsh destroy as we want a clean filesystem to avoid issues in the resize.

Get current image information

After the image has shutdown we can now go ahead and get some information on the disk configuration:

1
2
3
4
5
6

virt-filesystems --long --parts --blkdevs -h -a centos_centos6.qcow2

Name       Type       Size  Parent
/dev/sda1  partition  200M  /dev/sda
/dev/sda2  partition  9.8G  /dev/sda
/dev/sda   device     10G   -

As can be seen here there is a single 10GB virtual disk residing on /dev/sda

virt-rezise

We must then create a destination disk image, of the required total size

1

qemu-img create -f qcow2 outfile 150G

I have opted to use the –expand flag, if this is not specified a new partition is created to ocupy the free space, refer to man virt-resize for more advanced options such as splitting the freespace to grow existing partitions (i.e. expand the boot partition +100M)

1

virt-resize --expand /dev/sda2 original.qcow2 outfile.qcow2

Go make a coffee as this step will take a while to complete.

Finishing up

If you were to start the instance back up now using outfile.qcow2 as the disk image, you would find the OS reports the original disk size, this is due to the LVM configuration which we can not change “online” (unless of course you are changing a partition that can be unmounted, not the case here).

We will use guestfish to complete the process.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24

guestfish --rw -a outfile.qcow2

Welcome to guestfish, the libguestfs filesystem interactive shell for
editing virtual machine filesystems.

Type: 'help' for help on commands
      'man' to read the manual
      'quit' to quit the shell

><fs> run
><fs> list-filesystems
/dev/vda1: ext4
/dev/VolGroup00/LogVol00: ext4
/dev/VolGroup00/LogVol01: swap
><fs> lvresize-free /dev/VolGroup00/LogVol00 100
><fs> resize2fs /dev/VolGroup00/LogVol00
><fs> e2fsck-f /dev/VolGroup00/LogVol00
><fs> exit

virt-df -h outfile.qcow2
Filesystem                                Size       Used  Available  Use%
centos_el6_php53_lap:/dev/sda1            194M        52M       132M   27%
centos_el6_php53_lap:/dev/VolGroup00/LogVol00
                                          146G       1.1G       137G    1%

Your lvm configuration may differ change the above according to the output from list-filesystems.

Note: I run e2fsck-f as a precaution, this is not a required step though I highly recomend doing this.

Now finally swap out the images (or update the libvirt xml file, it’s up to you)

1
2
3

mv ./original.qcow2 ./original.bak
mv ./outfile.qcow2 ./original.qcow2
virsh start instance_name

If you instance starts successfully and all your data is intact the original.bak can be safely removed.

This is now 19 days since, and I realised I had not gotten around to writing a blog post on the project.

The Problem

The general public are unaware just how much data they send/receive at any given time; especially if said person has a “smart phone” the wealth of personal data a person carries around in their pocket can be staggering; more so that they have absolutely no clue how bad that can potentially be.

Want to carry out a little experiment?

1. Gather some none netsec aware people
2. How many of them can tell you right now without looking, if their phones wifi is enabled?
3. How many can do the same for bluetooth?
4. Without giving details, how many have passwords / bank details / something that shoudln’t be on their phone; on their phone?

You’ll be concerned with the results (unless you have somehow found a random grouping of people completely aware of their phones function and content at all times …).

Bridging the gap

In my experience no matter how you phrase it; for the general end consumer any conversation on netsec is met with indifference mainly due to a lack of understanding which is frustrating to say the least.

However you can can two directions in this situation berate the stupid luser; or you can attempt to educate them, and to that effect the most successfull method is something visual, in the form of a practical demonstration of the point you are trying to get across.

Why? It removes the need for the end consumer to attempt to mentally visualise what you are describing; all puns aside this makes it far easier for the end consumer to understand.

Education, got it … so why the pi?

Simple really, inexpensive 600MHZ arm processor that can boot linux and run from a battery pack.

The peak consumption I read somewhere is around 700ma, the battery pack in question is a 5000maH which asusming we see a 60% return on a full charge equestes to roughly 4.5hrs run time total.

1. Low power consumption
2. Easily portable
3. Relativly inexpensive
4. Runs linux

The Concept

I’d like to assume if you are reading this, you have at least a basic knowledge of netsec so at this point the post becomes less end user friendly …

1. Jassegar - utilizing jassegar to masquerade as a trusted ap, and route traffic through ethernet / usb 3g dongle. a. Desposable - setup and go, access data over 3g connection via a reverse tunnel; remote wipe / destruction.
2. Karma - I don’t know much about the karma quite, it appears this can be used for much the same as jassegar.

Couple the above with airdrop-ng for active denial of all wifi in the area, and suddenly every smart phone / laptop in the area is routing via the pwnberry pi and NO ONE is the wiser.

Proof or STFU

Whilst I don’t have a working demo at this time, perhaps some photos of the “build” would suffice?

Running left to right:

1. SD Card
2. 5000maH usb battery
3. Bottom of the “weather proof box”
4. Raspberry PI
5. ALFA awus036nh

All boxed in the “Premium” container, aka rubber sealed tupperware of doom …

To come …

I’m trying out different distributions to achieve this, pwnpi is looking promising at the moment.

As always time is limited so it’s on an as / when basis.

That said it has some great potential, if you’ve been following my open source contributions and posts you’ll known I have an special affinity for Openstack, Aeolus, and of course Opennebula.

As such I’ve taken to jumping in “feet first”, what better way eh?

Last October I was fortunate enough to attend the Openstack training in London, hosted by Rackspace, recently I now have a full openstack deployment running on fedora 17 on my laptop for prototyping, and testing (I have of course been bugreporting to redhat bugzilla! and I encourage you to do the same!).

I met some great people on the course last October, which unfortunatly I’ve only managed to keep in contact with a few of (if you’re reading this and were there get in contact!).

I have some upstream commits for: EPEL Openstack, libcloud, aeolus, boxgrinder … and I’ve gotten to a point this year where I can reflect, and make a post to that effect.

In short I have one problem with the cloud, and that’s the marketing; let me explain why, marketing is driven to make sales, it does not care about the education of the end user as to the product they are paying for, (and frankly hearing my parents / clients ask “Can’t you just use the cloud?” makes me want to break out the beating stick of education, more for the marketing people I belive in making a solution right for the indvidual not for the bottom line…), as I’ve come to know more on the systems involved it’s a revolution, now calm down and let me explain.

Yes the cloud is simply virtualization if you break it down into it’s rawest form, and that has been around for decades … but what “the cloud” is doing despite the marketing fluff, is comoditizing the technology and plating it firmly in the hands of users who have little to no technical background or knowlege, why is this a revolution?

Inherently a person who is somewhat intellegent is curious, curiosity (Despite killing the cat, though if my neighbours cat craps in my garden again it may well be my boot and not the curioisity) leads to discovery, this inturn leads to understanding; putting something so powerful so simply within reach of those who do not understand the technology both increases it’s proftiablity and should said end user persue their curosity they will learn.

Right so education for the massses, what’s next hugging trees? Not quiet the you may be missing the point, what’s better than an educated client someone who knows what they want and the potential technologies to achieve it as apposed to the uneducated who take the line of “it can’t be that hard all you do is sit there and tap the keyboard all day”.

There is a very real gap in understanding between the end user, and the Sysadmin/Devops supporting it, the cloud may well help to bridge the gap between the technology and the user, such as Devops bridges the gap between operations and the developer.

So, pulling this back to the original point of this blog, I appear to have gone off at a tangent.

1. I’ve conveted Wordpress -> Jekyll + Octopress
2. I’ve worked on the Rakefile to push differing assets to cloudfiles/
3. I am now just waiting on clouddns to allow CNAME records for the main domain, then …
4. saiweb.co.uk will exist purely in cdn.

With any luck I will be the first but this is reliant on the dns options becomming available, please comment and let me know your thoughts!

In this post I follow on from Setting up Nova and Glance, and now moving installing and Integrating keystone. I’d first like to give credit to IBM developerWorks the guys in #openstack @ freenode IRC, and Psycle Interactive without whom I would not of been able to complete this write up.

Please be aware the following applies to 2011.3 ONLY! (Diablo Final) the configuration to come in Essex is far simpler, if when reading this post your packages are 2012.X you have just installed essex and this is not relevant, anyway here we go …

1

yum install openstack-keystone

Keystone itself has it’s own tirade of concepts to get to grips with … tenant, user, role, service, token etc … I’m not going to go into detail on those concetps, for that Please see the documentation.

Configuring mySQL

First thing I am going to do is change from sqlite to mySQL connection, this involves editing line 54 of /etc/keystone/keystone.conf

1

sql_connection = mysql://keystone:keystone@localhost/keystone

Ignoring the default_store configuration at the top of the file, as this states sqllite, from what I can tell this simply instructs keystone to use the sqlAlchemy driver, which we just updated to point to mySQL.

Now like glance we need to restart keystone for the database to be populated.

1

service openstack-keystone restart

Now run keystone-manage with no args if you see

1
2
3

File "/usr/lib/python2.6/site-packages/keystone/manage/__init__.py", line 283, in main
    raise exc
sqlalchemy.exc.OperationalError: (OperationalError) (1044, "Access denied for user 'keystone'@'localhost' to database 'keystone'") None None

Review your keystone.conf file and ensure your mySQL credentials are correct, once done start keystone again.

Initial Credentials

Now we need to create an admin Tenant, and add an admin user to this tenancy.

1
2
3
4
5
6
7
8
9
10

keystone-manage tenant add adminTenant
SUCCESS: Tenant adminTenant created.
keystone-manage user add adminUser <password>
SUCCESS: User adminUser created.
keystone-manage role add Admin
SUCCESS: Role Admin created successfully.
keystone-manage role grant Admin adminUser
SUCCESS: Granted Admin the adminUser role on None.
keystone-manage role grant Admin adminUser adminTenant
SUCCESS: Granted Admin the adminUser role on adminTenant.

Ok so we have just:

1. setup a tenant named adminTenant.
2. setup a user named adminUser and specified their password.
3. created an admin role.
4. assigned the adminUser to the Admin role.
5. granted adminUser the Admin role to the adminTenant

Note: the outputs are a little confusion on the role assignments…

“Granted Admin the adminUser role on adminTenant”,

it appears the string output has the arguments in the wrong order here it should read:

“Granted adminUser the Admin role on adminTenant”.

I have however verified the mySQL data and can see the roles being correctly assigned.

Also the output from

1
2

keystone-manage role grant help
Missing arguments: role grant 'role' 'user' 'tenant (optional)'

Confirms the arguments are being entered in the correct order.

i.e.

mysql> select * from user_roles;
+----+---------+---------+-----------+
| id | user_id | role_id | tenant_id |
+----+---------+---------+-----------+
|  1 |       1 |       1 |      NULL |
|  2 |       1 |       1 |         1 |
+----+---------+---------+-----------+
2 rows in set (0.00 sec)

Now we need to configure keystone to recognise these new admin roles.

Lines 41 and 44:

1
2

keystone-admin-role = Admin
keystone-service-admin-role = KeystoneServiceAdmin

Edit these to reflect your Admin role accordingly and then restart openstack-keystone The above shows seperate roles for general and service admin, in my case I set these to the same role, it is of course entirely up to you and your delegation setup. If you choose to retain the KeystoneServiceAdmin delegation you will need to setup the role as per the Admin role above and run through the grants accordingly.

Setting up the Service token and service definitions

1
2

keystone-manage token add 999888777666 adminUser adminTenant 2012-12-23T00:00
SUCCESS: Token 999888777666 created.

If instead you get an error:

1
2
3
4
5
6
7
8

ERROR: 'NoneType' object has no attribute 'id'
2012-04-23 12:27:29    ERROR [root] 'NoneType' object has no attribute 'id'
Traceback (most recent call last):
  File "/usr/bin/keystone-manage", line 16, in <module>
    keystone.manage.main()
  File "/usr/lib/python2.6/site-packages/keystone/manage/__init__.py", line 283, in main
    raise exc
AttributeError: 'NoneType' object has no attribute 'id'

check your have correctly entered adminUser adminTenant (or the details you have entered) including correct capitilization.

1
2
3
4
5
6

keystone-manage service add nova compute "Openstack Compute Service"
SUCCESS: Service nova created successfully.
keystone-manage service add glance image "Openstack Image Service"
SUCCESS: Service glance created successfully.
keystone-manage service add keystone identity "Openstack Image Service"
SUCCESS: Service keystone created successfully.

Defining endPoints

Nova Here I managed to confuse myself, so let me be clear, this needs the nova_api service ip, not each compute node, meaning you only need one endpoint.

1
2

keystone-manage endpointTemplates add regionOne nova http://<nova_api_ip>:8774/v1.1/%tenant_id% http://<nova_api_ip>:8774/v1.1/%tenant_id% http://<nova_api_ip>:8774/v1.1/%tenant_id% 1 1
SUCCESS: Created EndpointTemplates for nova pointing to http://<nova_api_ip>:8774/v1.1/%tenant_id%

The 3 URL arguments are for publicURL, internalURL, adminURL (No idea if that is the order).

Glance

1
2

keystone-manage endpointTemplates add regionOne nova http://<glance_ip>:9292/v1 http://<nova_api_ip>:9292/v1 http://<nova_api_ip>:9292/v1 1 1
SUCCESS: Created EndpointTemplates for glance pointing to http://<glance_ip>:9292/v1

Keystone

1
2

keystone-manage endpointTemplates add pi-whc keystone http://<keystone_ip>:5000/v2.0 http://<keystone_ip>:5000/v2.0 http://<keystone_ip>:5000/v2.0 1 1
SUCCESS: Created EndpointTemplates for keystone pointing to http://<keystone_ip>:5000/v2.0.

Configuring Nova

Now we have keystone setup we need to configure nova to use keystone for authentication, by editing /etc/nova/api-paste.ini. Now there are seveal edits required, as such what follows are snippets of those changes.

EC2 Section modification

line 22 and 27 ([pipeline:ec2cloud] and [pipeline:ec2admin] sections).

1

pipeline = logrequest totoken authtoken keystonecontext ec2noauth cloudrequest authorizer ec2executor

New section for EC2 (in my config lines 60-61)

1
2

[filter:totoken]
paste.filter_factory = keystone.middleware.ec2_token:EC2Token.factory

Openstack section modification

Modification to [pipeline:openstackapi10] and [pipeline:openstackapi11] sections.

1
2
3
4
5

[pipeline:openstackapi10]
pipeline = faultwrap authtoken keystonecontext ratelimit extensions osapiapp10

[pipeline:openstackapi11]
pipeline = faultwrap authtoken keystonecontext ratelimit extensions osapiapp11

Shared section addition

We now need to add a complete new subsection to the .ini file

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17

##########
# Shared #
##########

[filter:keystonecontext]
paste.filter_factory = keystone.middleware.nova_keystone_context:NovaKeystoneContext.factory

[filter:authtoken]
paste.filter_factory = keystone.middleware.auth_token:filter_factory
service_protocol = http
service_host = <keystone_ip>
service_port = 5000
auth_host = <keystone_ip>
auth_port = 35357
auth_protocol = http
auth_uri = http://<keystone_ip>:5000/
admin_token = 999888777666

NOTE: you will want to change this to https, but I will not be covering https configuration in this post.

Check that your configuration is working:

1

curl -d '{"auth": {"tenantName": "adminTenant", "passwordCredentials":{"username": "adminUser", "password": "password"}}}' -H "Content-type: application/json" http://<keystone_ip>:35357/v2.0/tokens | python -mjson.tool

Now restart openstack-nova-api

1

service openstack-nova-api restart

Verifying nova keystone integration

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24

nova --debug --username=adminUser --apikey=<password> --url=http://<keystone_ip>:5000/v2.0 --version=1.1 list
connect: (<keystone_ip>, 5000)
send: 'POST /tokens HTTP/1.1\r\nHost: <keystone_ip>:5000\r\nContent-Length: 69\r\ncontent-type: application/json\r\naccept-encoding: gzip, deflate\r\nuser-agent: python-novaclient\r\n\r\n'
send: '{"passwordCredentials": {"username": "adminUser", "password": "<password>"}}'
reply: 'HTTP/1.1 400 Bad Request\r\n'
header: Content-Type: application/json; charset=UTF-8
header: Content-Length: 60
header: Date: Mon, 23 Apr 2012 14:16:13 GMT
Traceback (most recent call last):
  File "/usr/bin/nova", line 9, in <module>
    load_entry_point('python-novaclient==2.6.1', 'console_scripts', 'nova')()
  File "/usr/lib/python2.6/site-packages/novaclient/shell.py", line 209, in main
    OpenStackComputeShell().main(sys.argv[1:])
  File "/usr/lib/python2.6/site-packages/novaclient/shell.py", line 166, in main
    self.cs.authenticate()
  File "/usr/lib/python2.6/site-packages/novaclient/v1_1/client.py", line 54, in authenticate
    self.client.authenticate()
  File "/usr/lib/python2.6/site-packages/novaclient/client.py", line 140, in authenticate
    auth_url = self._v2_auth(auth_url)
  File "/usr/lib/python2.6/site-packages/novaclient/client.py", line 180, in _v2_auth
    resp, body = self.request(token_url, "POST", body=body)
  File "/usr/lib/python2.6/site-packages/novaclient/client.py", line 87, in request
    raise exceptions.from_response(resp, body)
novaclient.exceptions.BadRequest: Expecting auth (HTTP 400)

Don’t PANIC! it seems there was never a 2011.3 build for python-novaclient, as such we can “cheat” a little, and use 2012.1-1

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20

rpm -Uvh http://pbrady.fedorapeople.org/openstack-el6/python-novaclient-2012.1-1.el6.noarch.rpm
nova --debug --os_username=adminUser --os_password=<password> --os_tenant_name=adminTenant --os_auth_url=http://<keystone_ip>:5000/v2.0/ usage-list
connect: (<keystone_ip>, 5000)
send: 'POST /v2.0/tokens HTTP/1.1\r\nHost: <keystone_ip>:5000\r\nContent-Length: 110\r\ncontent-type: application/json\r\naccept-encoding: gzip, deflate\r\naccept: application/json\r\nuser-agent: python-novaclient\r\n\r\n'
send: '{"auth": {"tenantName": "adminTenant", "passwordCredentials": {"username": "adminUser", "password": "psycle"}}}'
reply: 'HTTP/1.1 200 OK\r\n'
header: Content-Type: application/json; charset=UTF-8
header: Content-Length: 924
header: Date: Mon, 23 Apr 2012 15:14:00 GMT
connect: (<nova_ip>, 8774)
send: u'GET /v1.1/1/os-simple-tenant-usage?start=2012-03-26T16:14:00.749451&end=2012-04-24T16:14:00.749491&detailed=1 HTTP/1.1\r\nHost: <keystone_ip>:8774\r\nx-auth-project-id: adminTenant\r\nx-auth-token: 999888777666\r\naccept-encoding: gzip, deflate\r\naccept: application/json\r\nuser-agent: python-novaclient\r\n\r\n'
reply: 'HTTP/1.1 200 OK\r\n'
header: Content-Type: application/json
header: Content-Length: 21
header: Date: Mon, 23 Apr 2012 15:14:00 GMT
Usage from 2012-03-26 to 2012-04-24:
+-----------+-----------+--------------+-----------+---------------+
| Tenant ID | Instances | RAM MB-Hours | CPU Hours | Disk GB-Hours |
+-----------+-----------+--------------+-----------+---------------+
+-----------+-----------+--------------+-----------+---------------+

You can also follow diablo more closely by using griddynamics’ rpm package

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21

rpm -e --nodeps python-novavclient
rpm -Uvh http://yum.griddynamics.net/yum/diablo/python-novaclient-2011.3-b2489.noarch.rpm
nova --debug --username adminUser --password <password> --tenant_name adminTenant --auth_url http://<keystone_ip>:5000/v2.0/ usage-list
connect: (<keystone_ip>, 5000)
send: 'POST /v2.0/tokens HTTP/1.1\r\nHost: <keystone_ip>:5000\r\nContent-Length: 110\r\ncontent-type: application/json\r\naccept-encoding: gzip, deflate\r\naccept: application/json\r\nuser-agent: python-novaclient\r\n\r\n'
send: '{"auth": {"tenantName": "adminTenant", "passwordCredentials": {"username": "adminUser", "password": "<password>"}}}'
reply: 'HTTP/1.1 200 OK\r\n'
header: Content-Type: application/json; charset=UTF-8
header: Content-Length: 924
header: Date: Mon, 23 Apr 2012 15:27:01 GMT
connect: (<nova_ip>, 8774)
send: u'GET /v1.1/1/os-simple-tenant-usage?start=2012-03-26T16:27:01.859467&end=2012-04-24T16:27:01.859524&detailed=1 HTTP/1.1\r\nHost: <keystone_ip>:8774\r\nx-auth-project-id: adminTenant\r\nx-auth-token: 999888777666\r\naccept-encoding: gzip, deflate\r\naccept: application/json\r\nuser-agent: python-novaclient\r\n\r\n'
reply: 'HTTP/1.1 200 OK\r\n'
header: Content-Type: application/json
header: Content-Length: 21
header: Date: Mon, 23 Apr 2012 15:27:01 GMT
Usage from 2012-03-26 to 2012-04-24:
+-----------+-----------+--------------+-----------+---------------+
| Tenant ID | Instances | RAM MB-Hours | CPU Hours | Disk GB-Hours |
+-----------+-----------+--------------+-----------+---------------+
+-----------+-----------+--------------+-----------+---------------+

BE WARNED

Most of the other commands for myself are presently returning 404 / 500 errors, with the Essex Release Impending the current EPEL advice seems to be to use Essex, I will update as/when I can with futher information on these issues.

For instance on a: flavor-create a 500 error is encountered with the following logged in api.log

1
2
3

...
(nova.api.openstack): TRACE: AttributeError: 'ControllerV11' object has no attribute 'create'
...

Configuring Glance

Modify /etc/glance/glance-api.conf

Comment out line 138 and uncomment 140

1
2
3
4

[pipeline:glance-api]
#pipeline = versionnegotiation context apiv1app
# NOTE: use the following pipeline for keystone
pipeline = versionnegotiation authtoken auth-context apiv1app

Modify lines 165-174 accordingly

1
2
3
4
5
6
7
8
9
10

[filter:authtoken]
paste.filter_factory = keystone.middleware.auth_token:filter_factory
service_protocol = http
service_host = <keystone_ip>
service_port = 5000
auth_host = <keystone_ip>
auth_port = 35357
auth_protocol = http
auth_uri = http://<keystone_ip>:5000/
admin_token = 999888777666

now edit /etc/glance/glance-registry.conf and again comment out the current pipline= line and uncomment the keystone line.

1
2
3
4

[pipeline:glance-registry]
#pipeline = context registryapp
# NOTE: use the following pipeline for keystone
pipeline = authtoken auth-context registryapp

Update the authtoken filter accordingly

1
2
3
4
5
6
7
8
9
10

[filter:authtoken]
paste.filter_factory = keystone.middleware.auth_token:filter_factory
service_protocol = http
service_host = <keystone_ip>
service_port = 5000
auth_host = <keystone_ip>
auth_port = 35357
auth_protocol = http
auth_uri = http://<keystone_ip>:5000/
admin_token = 999888777666

Restart glance

1
2
3
4
5

for i in api registry; do service openstack-glance-$i restart; done
Stopping openstack-glance-api:                             [  OK  ]
Starting openstack-glance-api:                             [  OK  ]
Stopping openstack-glance-registry:                        [  OK  ]
Starting openstack-glance-registry:                        [  OK  ]

testing Keystone

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18

nova --debug --username adminUser --password <password> --tenant_name adminTenant --auth_url http://<keystone_ip>:5000/v2.0/ image-list
connect: (<keystone_ip>, 5000)
send: 'POST /v2.0/tokens HTTP/1.1\r\nHost: <keystone_ip>:5000\r\nContent-Length: 110\r\ncontent-type: application/json\r\naccept-encoding: gzip, deflate\r\naccept: application/json\r\nuser-agent: python-novaclient\r\n\r\n'
send: '{"auth": {"tenantName": "adminTenant", "passwordCredentials": {"username": "adminUser", "password": "<password>"}}}'
reply: 'HTTP/1.1 200 OK\r\n'
header: Content-Type: application/json; charset=UTF-8
header: Content-Length: 924
header: Date: Mon, 23 Apr 2012 15:48:56 GMT
connect: (<nova_ip>, 8774)
send: u'GET /v1.1/1/images/detail HTTP/1.1\r\nHost: <keystone_ip>:8774\r\nx-auth-project-id: adminTenant\r\nx-auth-token: 999888777666\r\naccept-encoding: gzip, deflate\r\naccept: application/json\r\nuser-agent: python-novaclient\r\n\r\n'
reply: 'HTTP/1.1 200 OK\r\n'
header: Content-Type: application/json
header: Content-Length: 14
header: Date: Mon, 23 Apr 2012 15:48:56 GMT
+----+------+--------+--------+
| ID | Name | Status | Server |
+----+------+--------+--------+
+----+------+--------+--------+

More to follow soon as I work through these issues, and later move onto 2012.X (Essex)

layout: post title: “yummy chroots. Building chroots with yum on fedora 16” date: 2012-03-07 16:59 comments: true published: true categories:

- linux

We’re going to build a minimal chroot directory for Fedora 16 using yum and rpm, we are using the ChrootDirectory functionality of Openssh which only came in >= 4.9p1

Credit goes http://prefetch.net/articles/yumchrootlinux.html for a great article getting me started on this.

As root:

mkdir --mode=700 -p /chroot/chrootuser
rpm --root /chroot/chrootuser --initdb
yumdownload --destdir=/var/tmp fedora-release
rpm --root /chroot/chrootuser -ivh --nodeps /var/tmp/fedora-release*rpm
yum --installroot=/chroot/chrootuser -y install bash
yum --installroot=/chroot/chrootuser -y install coreutils
groupadd chrooted

Edit /etc/ssh/sshd_config

Match Group chrooted
        ChrootDirectory /chroot/%u
        AllowTcpForwarding no
        X11Forwarding no
        AllowAgentForwarding no
        PermitRootLogin no
        ForceCommand /bin/bash

And restart the service: systemctl restart sshd.service

Add the user:

useradd -G chrooted -d /chroot/chrootuser chrootuser

ssh in as the user and they will be in the jailed directory

1. You have a mysql instance installed and running
2. You have a rabbitmq-server installed and running
3. You have kvm installed and running (libvirt)
4. You have selinux set to permissive, as I will not be covering selinux rules here at this time and I do not think disabled is a valid option ;-)

I will also be carrying out mySQL configuration of glance and nova, for 2011.3 (Diablo), though most if not all of this should be portable to the Essex release

Install EPEL

rpm -Uvh http://download.fedoraproject.org/pub/epel/6/i386/epel-release-6-5.noarch.rpm

Install Nova and Glance

yum -y install openstack-nova openstack-glance

yum should take care of all the dependencies here, and install both with a minimal configuration.

Burning and Rebuilding bridges

First thing’s first KVM is going to install with it’s own default bridged networking, this provides NAT.

Which is also noted as being very slow (There is/was an note on the wiki@ linux-kvm.org but I have been unable to locate it at the time of writing)

If you are only setting this up for experimentation you can run with the default networking, simply use vibr0 in your nova.conf instead of br0, and ensure you have ipv4 forwarding enabled.

Burning Bridges

1
2
3
4
5
6
7
8
9

virsh net-list
Name                 State      Autostart
-----------------------------------------
default              active     yes 
virsh net-destroy default
Network default destroyed
virsh net-undefine default
Network default has been undefined
service libvirtd restart

Building Bridges

The theory here is that this configuration of bridge will give us near native network performance, which if you are setting up for use beyond a throwaway sandbox, you really do not want to start introducing bottlenecks.

Shutdown and disable NetworkManager

1
2
3

service NetworkManager stop
chkconfig NetworkManager off
chkconfig network on

If you know of a NetworkManager friendly way of doing the following please let me know!

In this scenario br0 becomes your current eth0

/etc/sysconfig/network-scripts/ifcfg-br0

1
2
3
4
5
6
7
8

DEVICE=br0
TYPE=Bridge
BOOTPROTO=static
IPADDR=192.168.99.1
NETMASK=255.255.255.0
GATEWAY=192.168.99.254
ONBOOT=yes
DELAY=0

/etc/sysconfig/network-scripts/ifcfg-eth0

1
2
3
4
5
6
7

DEVICE=eth0
BOOTPROTO=none
TYPE=Ethernet
HWADDR=00:11:22:33:44:55
ONBOOT=yes
USERCTL=no
BRIDGE=br0

There is plenty more fun to be had here such as bonded interfaces (I myself have a few systems with bonded interfaces as such becoming br0 -> bond0 -> NIC’s), but that’s for another time.

Note: you may also use brctl for temporary configurations if you are just experimenting.

Caution: my network dropped out immediatly on my testbox, most likely because networkmanager was running, always ensure you can attach to the head of your box when doing network configuration ;-)

Once you have these configurations in place (Ensuring your have replaced the placeholder IP’s and MAC address with valid ones) you can now go for a

1

service network restart

All being well you’ll lose and re-establish connection, of you’ll be attaching a monitor / to kvm over ip.

Configuring Nova

First we’re going to need a blank database, please ensure you change the placeholder password that follows for something more secure, and amend the host if you are using mySQL on the same host as nova.

create database nova;
grant all privileges on nova.* to 'nova'@'localhost' identified by 'nova';

Your /etc/nova.conf should resemble this:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16

--logdir=/var/log/nova
--state_path=/var/lib/nova
--lock_path=/var/lib/nova/tmp
--dhcpbridge=/usr/bin/nova-dhcpbridge
--dhcpbridge_flagfile=/etc/nova/nova.conf
--injected_network_template=/usr/share/nova/interfaces.template
--libvirt_xml_template=/usr/share/nova/libvirt.xml.template
--vpn_client_template=/usr/share/nova/client.ovpn.template
--credentials_template=/usr/share/nova/novarc.template
--network_manager=nova.network.manager.FlatDHCPManager
--iscsi_helper=tgtadm
--sql_connection=mysql://nova:nova@localhost/nova
--rabbit_host=localhost
--glance_api_servers=localhost:9292
--iscsi_ip_prefix=10.0.0.1
--bridge=br0

Setup the database and start the relevant nova services

nova-manage db sync
for i in api network scheduler compute; do service openstack-nova-$i start; done
for i in api network scheduler compute; do chkconfig openstack-nova-$i on; done

Note: you could also use openstack-nova-db-setup instead of “nova-manage db sync”, but it requires mysql-server, which at the time of writing if you have Percona installed will falsely adivse you a need to install mysql-server, Percona need to add: “Provides: mysql-server” to their spec ideally.

Remember this is only a basic setup so a lot of the options are left default such as the network_manager, I will cover their options at a later date.

Onto setting up a basic user (Note: this will be replaced in future posts with keystone)

1
2
3

nova-manage user admin saiweb
nova-manage project create saiweb saiweb
nova-manage network create saiweb 192.168.99.1/24 1 256 --bridge=br0

Take a moment to run a quick check on your services and network

1
2
3
4
5
6
7
8
9

nova-manage service list
Binary           Host                                 Zone             Status     State Updated_At
nova-network     oneiroi                              nova             enabled    :-)   2012-03-07 22:21:10
nova-compute     oneiroi                              nova             enabled    :-)   2012-03-07 22:21:12
nova-scheduler   oneiroi                              nova             enabled    :-)   2012-03-07 22:21:10

nova-manage network list
id      IPv4                IPv6            start address   DNS1            DNS2            VlanID          project         uuid           
1       10.0.0.0/24         None            10.0.0.2        8.8.4.4         None            None            None            7d480f13-47f7-4117-9889-d44f378c3fee

Now we need the nova credentials for this user + project.

1
2
3
4
5
6
7
8
9
10

nova-manage project zipfile saiweb saiweb
unzip nova.zip
mv ./{novarc,pk.pem,cert.pem,cacert.pem} ~/.nova/
chmod 700 ~/.nova
chmod 600 ~/.nova/*
rm ./nova.zip
echo ". ~/.nova/novarc" >> ~/.bashrc
source ~/.bashrc
euca-add-keypair nova_key > ~/.nova/nova_key.priv
chmod 600  ~/.nova/nova_key.priv

Configuring Glance

The only change I made was to make glance use mysql.

create database glance;
grant all privilges on glance.* to 'glance'@'localhost' identified by 'glance';

/etc/glance/glance-resgistry.conf

1
2
3

...
sql_connection = mysql://glance:glance@localhost/glance
...

Once you have made the change, unlike nova all you need do is start glance and it will setup the database.

for i in api registry; do chkconfig openstack-glance-$i on; service openstack-glance-$i start; done

Now were going to need an image, I’m using the BT5-R2 .iso as an example, you could use any of the pre-generated images out there, or even build them using oz

1

glance add name="BT5-R2-Gnome-x64" is_public=True container_format=ovf disk_format=raw < ./BT5R2-GNOME-64.iso

Once the import has completed it should appear in your glance index

1
2
3
4

glance index
ID               Name                           Disk Format          Container Format     Size          
---------------- ------------------------------ -------------------- -------------------- --------------
1                BT5-R2-Gnome-x64               raw                  ovf                      2762084352

And assuming you setup your nova.conf correctly you should now be able to see this image from nova

1
2
3
4
5
6

nova image-list
+----+------------------+--------+
| ID |       Name       | Status |
+----+------------------+--------+
| 1  | BT5-R2-Gnome-x64 | ACTIVE |
+----+------------------+--------+

You will also have some default instance sizes aka flavours (commands use american spelling flavor).

1
2
3
4
5
6

nova-manage flavor list
m1.medium: Memory: 4096MB, VCPUS: 2, Storage: 40GB, FlavorID: 3, Swap: 0MB, RXTX Quota: 0GB, RXTX Cap: 0MB
m1.large: Memory: 8192MB, VCPUS: 4, Storage: 80GB, FlavorID: 4, Swap: 0MB, RXTX Quota: 0GB, RXTX Cap: 0MB
m1.tiny: Memory: 512MB, VCPUS: 1, Storage: 0GB, FlavorID: 1, Swap: 0MB, RXTX Quota: 0GB, RXTX Cap: 0MB
m1.xlarge: Memory: 16384MB, VCPUS: 8, Storage: 160GB, FlavorID: 5, Swap: 0MB, RXTX Quota: 0GB, RXTX Cap: 0MB
m1.small: Memory: 2048MB, VCPUS: 1, Storage: 20GB, FlavorID: 2, Swap: 0MB, RXTX Quota: 0GB, RXTX Cap: 0MB

Booting your first Instance

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27

nova boot --flavor 2 --image 1 "BT5"
+--------------+--------------------------------------+
|   Property   |                Value                 |
+--------------+--------------------------------------+
| accessIPv4   |                                      |
| accessIPv6   |                                      |
| adminPass    | pnFKeVPpbb7bKKy6                     |
| config_drive |                                      |
| created      | 2012-03-07T23:11:59Z                 |
| flavor       | m1.small                             |
| hostId       |                                      |
| id           | 1                                    |
| image        | BT5-R2-Gnome-x64                     |
| key_name     | None                                 |
| metadata     | {}                                   |
| name         | BT5                                  |
| progress     | 0                                    |
| status       | BUILD                                |
| tenant_id    | saiweb                               |
| updated      | 2012-03-07T23:11:59Z                 |
| user_id      | saiweb                               |
| uuid         | fb08be47-2647-4cb2-86d8-867ea0ef4981 |
+--------------+--------------------------------------+
virsh list
 Id Name                 State
----------------------------------
  1 instance-00000001    running

And as iso-boot is not currently complete, this example falls down here, as the instance fails to boot from the .iso file, still you now have

1. Successfully configured nova
2. Sucessfully configured glance
3. Have nova using glance

All you need do is load a valid image into glance and boot using nova, so now I will be cheating a little I will create a blank 10GB qcow2 image, import it into glance boot it and use virt-manager to attach the .iso and reboot.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27

qemu-img create -f qcow2 blank.qcow2 10G
Formatting 'blank.qcow2', fmt=qcow2 size=10737418240 encryption=off cluster_size=65536
glance add name="blank-10G" is_public=True container_format=bare disk_format=qcow2 < ./blank.qcow2
Added new image with ID: 2
nova boot --flavor 2 --image 2 "BT5"
+--------------+--------------------------------------+
|   Property   |                Value                 |
+--------------+--------------------------------------+
| accessIPv4   |                                      |
| accessIPv6   |                                      |
| adminPass    | H3khDYMwheNNWBV3                     |
| config_drive |                                      |
| created      | 2012-03-07T23:01:50Z                 |
| flavor       | m1.small                             |
| hostId       |                                      |
| id           | 2                                    |
| image        | blank-10G                            |
| key_name     | None                                 |
| metadata     | {}                                   |
| name         | BT5                                  |
| progress     | 0                                    |
| status       | BUILD                                |
| tenant_id    | home                                 |
| updated      | 2012-03-07T23:01:50Z                 |
| user_id      | oneiroi                              |
| uuid         | 05ce2b5d-d03c-442e-99e3-2c079469ec5b |
+--------------+--------------------------------------+

Now I cheat I used virt-manager to force off the insance, create and attach an IDE cdrom and set it as the primary boot device. BT5 boots from the ISO and I can even begin to work through the install to hard drive menus, which as irony would have it prompts me that it needs an 11.5GB partition to install upon :D

I will cover producing proper images in my next openstack post, as the size of the storage volume should not be defined by the image in glance, it should be defined by the falvour being started.

n2n is a layer-two peer-to-peer virtual private network (VPN) which allows users to exploit features typical of P2P applications at network instead of application level. This means that users can gain native IP visibility (e.g. two PCs belonging to the same n2n network can ping each other) and be reachable with the same network IP address regardless of the network where they currently belong. In a nutshell, as OpenVPN moved SSL from application (e.g. used to implement the HTTPS protocol) to network protocol, n2n moves P2P from application to network level.

So why do I care ?

Some services you may wish to run on a public cloud such as Gluster do not have (at the time of writing) internal TLS (read: encryption), this n2n allows you to establish peer to peer vpn connections, wihtout the need of a single routing device (with some assume caveats I will cover shortly).

So in short you can have your own private network within the cloud environment without affecting that environment, this allows for:

1. TLS for services otherwise sent “in the clear”
2. Potential for Cluster services and floating IP’s without touching the host network infrastructure.

Installation

We are going to use EPEL, why? because I’m a packager and I will be using redhat for this setup, so admitedly I am a little biased toward RedHat, that said the majority of the following configurations should be portable to other distros, leave a commment if you get stuck I will try to help!

yum -y install n2n

And no I’m not using sudo i.m.o sudo is akin to “training wheels”, and somethign I will only generally use if I have too (such as maintaining an auditable system), you are of course welcome to use sudo yourself, I use “throw away” vm’s for all my experimentation so in these cases the ethos is if it’s broken it gets rebuilt.

SuperNode Setup

First thing’s first we’re going to need at least 1 Supernode, as I uderstand it a Supernode is used to register new peers and to retrieve currently connected peers. Once this list is retrieved the individual nodes will communicate directly (p2p), and not via the supernode.

Caveats to note:

1. If all supernodes are down, only existing peers can communicate, new peers can not.

supernode whilst installed does not at the time of writing provide an init.d/sysvinit script, you may use the following:

place the above in /etc/init.d/supernode and chmod +x i.e.

curl -o /etc/init.d/supernode https://raw.github.com/gist/1986260/b66b38da265ea14aac8d0ef7196a9ba98939716c/supernode.sh && chmod +x /etc/init.d/supernode

(Though I really do recommend you read through this code first before trusting it blindly!)

Note: Annoyingly I had to use the -f (foreground) flag to allow the daemon wrapper to function correctly with this process, there is more than likely a better solution, please feel free to revise the gist it is public.

Now as I have opted to use a non existant n2n account to daemonize the process this will need creating as will the pid directory.

useradd -d /dev/null -s /sbin/nologin n2n
mkdir /var/run/supernode && chown n2n:n2n /var/run/supernode

You will now be able to start your supernode with: /etc/init.d/supernode start.

In my configuration above I have chosen to bind port 1200, you can change this to any port, but remember that your vpn peers will need to be able to access this port. As such you will need the relevant iptables rules

iptables -N N2N
iptables -I INPUT -j N2N
iptables -A N2N -s <vpn peer> -p udp --dport 1200 -j ACCEPT

I highly recomend you limit your firewall to only allow connection from known peers, and that this is done over the internal interface (for which you do not generally pay bandwidth charges).

I also recomend you repeat this process on a 2nd node to provide 2 Supernodes (The maximum allowable) for greater resilliance.

Edge Setup

I have opted for a .conf file approach here, you can of course opt to instead embed everything in the sysvinit script.

DEVICE="n0"
ADDRESS="127.16.0.1"
MAC="00:11:22:33:44:55"
COMMUNITY="N2N"
SHAREDKEY="asdf12345"
SUPER1="1.2.3.4:1200"
SUPER2="1.2.3.5:1200"
PORT="1201"

Place this in /etc/edge.conf, you can negate ADDRESS if you wish to use DHCP, whilst you can also Negate SUPERNODE2 and MAC I do not recomend doing so for the following reasons.

1. Negating Supernode2 means there is only 1 supernode and as such a single point of failiure in the setup
2. Negating MAC is valid, however on loss of connection and restoration a new MAC is generated meaning all existing nodes can not communicate with the restored node untill their local ARP caches are cleared, specifiying a static MAC address ensures immediate restoration of communication.
3. I have made PORT a requirement, it is technically optional but fixing the port makes your iptables / firewall rules far easier.
4. Make sure you actually edit the file and replace the args with VALID ones, especially the SHAREDKEY as the above is in no way secure!
5. Make sure your ip and mac addresses are unique!

We need to prep the pid dir again:

mkdir /var/run/edge && chown n2n:n2n /var/run/edge

place the above in /etc/init.d/edge and chmod +x i.e.

curl -o /etc/init.d/edge https://raw.github.com/gist/1986260/3061d0fb9d6f2ddf1608f01917129d65b8131d33/edge.sh && chmod +x /etc/init.d/edge

(Again I HIGHLY recommend you actually read the code before blindly trusting it!)

Note: the –user option is negated in this init file. This is because we need to actually create a network interface, something that can only be done as root. As such we are reliant on the edge binary to drop privileges itself by providing the -u and -g arguments, these are of course assuming you have allready setup the n2n user, as per above and not just skipped to this section.

Add the Services and set them to run

chkconfig --add supernode
chkconfig --add edge
chkconfig supernode on
chkconfig edge on

Modify other services that are reliant on the VPN

Modify the “Requires” line in the sysvinit script for each service you want to only start once your VPN has been established.

...
# Required-Start: $local_fs $network $supernode $edge
...

Note: Whilst I have opted for requiring supernode here, you do not need this, you can require just your edge service, as the supernode does not have to run on the same device.

You should now be able to reboot and see all required services start up in the correct order.

And done, that’s where I am ending this blog post,

1. we have setup n2n with supernodes and edge
2. generated valid sysvinit scripts

expect future posts to cover more advanced n2n configuration as I discover the options available.

Why you may ask? Performance!

To facilitate running wordpress on the smalest possible CloudServer I am using Varnish which using Apache as the backend, now with static files I can get all the blogging functionality without the need for Wordpress nor varnish, yet still uncached content can lead to increased load on the server.

Also wordpress does not lend itself to scalability especially with the at the time of writing schema and sql queries (percona-query-advisor flags up a few wordpress core sql queries as non scalable).

But this does not mean you need remove the option of using Wordpress, for you client for instance keeping wordpress in place can aid in content generation simply through ease of use.

1. wordpress used as normal
2. wordpress content is ported to markdown
3. markdown used to generate html

Now this does have caveats:

1. you’re going to need to maintain designs in wordpress templates and markdown (_layouts).
2. you’re going to need to handel any shortcode plugins in your export process.
3. you’re going to need to handel any other content modifying plugins in your export process.
4. any UGC / Dynamic requests will still need PHP.

But it essentially replaces the whole caching layer with static content which then can be pushed to CDN.

And with CDN’s now supporting index files (S3, and Coming Soon @ CloudFiles) in essence entire sites can be placed on CDN whilst maintaining ease of content generation.

Now don’t get me wrong, this requires a whole lot of “glue” to get working, but the potential for serving an entire web app from CDN without Origin pull / cache headers etc, saves a lot of systems time in scaling and adressing performance issues, or rather makes them “less critical” as the “business” part of the webapp is all on CDN.

I’m still getting to grips with Jekyll and by extention Octopress to see what it can achieve, so expect more posts.

Well I for one say if that is the case then no one should be a “DevOps” without a background in Systems administration … let me explain.

Primarily I work with redhat rpm based systems for web application hosting at what I’d call an advanced level stracing, calling on linux c api’s as needed, fixing packaged and upstreaming the fixes, bug reporting etc (In my opinion something anyone using Opensource in their business should be doing!), I’m not going to go into complete detail on the tools and how I use them on a day to day basis as this moves from the point of this post entirely (that and it would take FAR too long to write …)

I also as part of my job I work in python, ruby, php, bash, tcl, c, c++, whatever tool is needed to do the job, let me say that again for clarity whatever tool is needed to do the job.

I could be a DBA, Sysadmin, TechSupport, Pentester at any given point of the day.

I analyse and profile web applications then go on to design hosting solutions for said applications.

I promote the use of SCM (Git in particular), unit testing and I’ve begun looking at Continuous integration methodologies.

I’m a commiter on the EPEL Openstack packages (Admittedly not as often as I would like at the moment … deadlines …), I also have upstream commits for libcloud and boxgrinder.

I work to the ethos that downtime is not acceptable, EVER! And if that means I have to profile, bugfix and code to ensure that is not the case then I will, I call it adapting and not being rigid.

I am presently looking at Chef to compliment my planned deploy of Openstack, for which I will be writing the configurations, this will in turn allow the development team to get on with their jobs, I already use kickstarts for my KVM deployments, Chef seems like the next logical step.

And whilst “The Cloud” has met with my skepticism, this is more to do with the over marketing claiming it is the solution to all your aliments … once you get past all the marketing fluff it is the way forward, and has been as such since a long time before “The Cloud” fluff came along.

So in short, I’m a Systems Administrator and I work damned hard to ensure those systems I administer stay online, if that means I need to work as a Developer, Pentester etc … then I will.

Whilst I can see that Devops in its current form could be stand alone from Systems Administration, it shouldn’t be …

You should not carry out Devops without a knowing the platforms you are deploying to, it’s like being a Cardiologist having spent 20 minutes on Operation (Yes overly melodramatic metaphor, remember uptime for me is that important.)

So what does that make me? aside from an overly paranoid uptime chanting nutter?

On Another note Saiweb.co.uk is 7 years old 26/03/2012 … I should really add more content …

What I did find a little lacking on the documentation side was the SCM integration (read: Source Control Management), git/svn etc …

In short so long as your rpm spec file is in your SCM (and it should be), moc will build your rpm from your sources in scm, which can be used for.

1. bleeding edge builds for testing
2. builds from “stable tags”

Yes yes yes … obvious I know …

So with no futher ado here is the syntax:

mock -r your_target --scm-enable --scm-option method=git --scm-option package=git_project --scm-option git_get='git clone git@git_ip_address:SCM_PKG.git SCM_PKG' --scm-option spec='SCM_PKG.spec' --scm-option branch=1-2 --scm-option write_tar=True -v

1. scm-enable - turns on the use of scm
2. scm-option - set an option for the scm in use

The above worked for me, you will need to adjust it acordingly, i.e. if your spec file is not named identically to that of your git project: –scm-option spec=’specfile_name.spec’

This will tie me over untill I get chance to play with my monkey farm

Firs you must meet the following conditions:

1. You are running gluster >= 3.0 <= 3.2 (May also work on 2.x I have not tested, and will not work with future versions if gluster change their use of xattrs)
2. You are running a replicated volume (Again I have not tested distributed volumes, in theory remove, re-add and rebalance will fix these)
3. You have a “good” copy of you data (This is essential this assume you have at least 1 brick with a good copy of the file system

Restrain and restore the “bad” brick

1. Shutdown all services that are using the mounted filesystem (i.e. httpd / nginx / *ftpd)
2. Unmount all the file systems on the node (glusterfs / nfs / etc …)
3. Grab a copy of stripxattr.py make sure you READ the README for installation requirements and usage
4. Run stripxattr.py against the backing filesystem on the “bad” node ONLY NOT AGAINST A GLUSTER MOUNT
5. From the “good” node, not rsync the data: rsync -gioprtv –progress /path/to/filesystem root@:/path/to
6. From the “good” node, trigger an ”auto heal” this will re-populate the xattr data (this must be done on a glusterfs mount not nfs/cifs/etc…)
7. Download listxattr.py once the self heal has completed see the README file for a “quick and dirty” consistency check
8. All being well you have now resolved a split-brain and can return your node to service

Current known gluster issues

1. NFS is much (48x in tests) faster for small files i.e. php webapps, but does not support distributed locking meaning: all nodes can write to the same file at the same time, this is what cause our original split brain

So what is the resolution int his case?

Selective use, use glusterfs for filesystems that you need distributed locking, often in large production deploys php files will not change often, in this case NFS is perfect.

If you are still writing php sessions to a file system then STOP IT and use a database! (Better yet use memcache).

There’s been a lot developing over the last few months, Openstack being one of my main focuses along with overhauling and provision new internal systems for Openstack to run upon, I have a plan so to speak …

I have some Openstack posts coming I just need to ensure that all parties are happy with me posting the information “in the clear” so to speak.

In one of the worst customer service experiences I’ve ever had the miss fortune to have been a part of all access was closed to our in office version control systems due to “high usage”, now this is a pretty essential service as you might imagine, how then to restore access, when the restrictions are beyond your control? (And I mean EVERY inbound port was dead …)

Fortunately it would seem outbound SSH was not affected, so after much vocal drawing and re-drawing many times over on the whiteboard I had a cunning plan …

Using 3 linux devices I would create the following.

1. A device through which using host entries / dns changes the version control would be available whilst not actually running on the box itself.

2. An in house device which would be the device on which the tunnels are created in the first place.

3. The device(s) on which the version control systems reside.

Gateway device

On the gateway device sshd_config needs to be updated with:

GatewayPorts yes

And sshd reloaded.

Also if you are using a local firewall (i.e. iptables) you will need to setup the appropriate rules as if the service were running natively on the device

Pivot Device

Generate rsa ssh keys and deploy your id_rsa.pub to the gateway device, (update sshd_config to enable RSA Auth if required)

The tunnel.

ssh <Gateway Device> -l root -g -N -R 0.0.0.0:<Service Port>:10.0.0.1:<Service Port>  -vvv

Now you only really need to use root if the port you need to gateway is a privileged port (<1024).

Here 10.0.0.1 is the internal address of the device the connection should “pivot” onto.

Once the tunnel was in place the services could be reached via the gateway device, this was essentially a “poor mans” NAT in a time of need, I really do not suggest this for long term use.

for i in {60..200}; do ping -c 1 -W 1 192.168.1.$i > /dev/null; ([[ $? == 0 ]] && echo "$i UP" || echo "$i DOWN");  done
1 UP
2 DOWN
3 UP
...

Note: for OSX use “ping -c 1 -t 1”

Chaining “UP” hosts for a quick (syn) port scan

for i in {60..200}; do ping -c 1 -W 1 192.168.1.$i > /dev/null; ([[ $? == 0 ]] && nc -v -n -z -w1 192.168.1.$i 20-22); done
(UNKNOWN) [192.168.1.1] 22 (ssh) open
(UNKNOWN) [192.168.1.3] 22 (ssh) open

Recover from a bad mysql password set (Update mysql.users set password=’Iforgotawherestatemenlulz’)

Assumes for every user there is an @localhost host, grabs the in memory password hash and resets

mysql -Bse 'Select distinct(user) from mysql.user;' | while read uname; do mysql -Bse "show grants for '$uname'@'localhost';" 2>&1 | grep IDENTIFIED | grep -v 'root' | grep -v 'ERROR' | sed 's|GRANT USAGE ON *.* TO ||g' | sed "s|@'localhost' IDENTIFIED BY PASSWORD||g" | awk '{print "Update user set Password="$2" where User="$1";"}' | mysql mysql; done

If you’ve run FLUSH PRIVILEGES; however you == b0ned.

Quick substitute and run

Command1:

ping -c 1 -t 1 192.168.1.1

Opps that’s OSX synatx

Command2:

^-t 1^-W 1

et voila corrected syntax.

Shortcuts

!! - Execute last command !ping - Execute last ping command, can be used to !any command just be careful. ctrl+r - reverse search, just start typing the cmd for it to search your history, hit tab to complete ctrl+a - jump to beginning of line ctrl+e - jump to end of the line

cURL FU

curl -I -L blahblah.tld - Run a HEAD and follow redirects (very handy for quicklooking @ bit.ly short URLS before hitting them in a browser).

python FU

python -m SimpleHTTPServer - serves the current pwd as a browseable directory (Very cool but VERY insecure) python -m cProfile script.py - generate trace stats for a script execution (Very handy for finding excessive loops)

DNS Fu

Wikipedia over DNS:

host -t txt fu.wp.dg.cx

fu.wp.dg.cx descriptive text “Fu may refer to: Fu (Technology, especially computer related) (used as a suffix) - relating to a person - Possessing superior skills in an art\; relating to an artifact - representing an expression of high art. code-fu, Perl-fu, C-fu, etc, Fu (literature),” ” a Chinese genre of rhymed prose, Fu (kana), a symbol in Japanese syllabaries, Fu County, in Shaanxi, China, Fu Foundation… http://a.vu/w:Fu”

Useful on some public wifi connections if you just want to look something up quick (dns is not always re-written).

Get all MX servers for a domain:

dig google.co.uk MX

; <<>> DiG 9.6.0-APPLE-P2 <<>> google.co.uk MX ;; global options: +cmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 64165 ;; flags: qr rd ra; QUERY: 1, ANSWER: 4, AUTHORITY: 4, ADDITIONAL: 4

;; QUESTION SECTION: ;google.co.uk. IN MX

;; ANSWER SECTION: google.co.uk. 10800 IN MX 10 google.com.s9a1.psmtp.com. google.co.uk. 10800 IN MX 10 google.com.s9a2.psmtp.com. google.co.uk. 10800 IN MX 10 google.com.s9b1.psmtp.com. google.co.uk. 10800 IN MX 10 google.com.s9b2.psmtp.com.

;; AUTHORITY SECTION: google.co.uk. 59925 IN NS ns2.google.com. google.co.uk. 59925 IN NS ns3.google.com. google.co.uk. 59925 IN NS ns4.google.com. google.co.uk. 59925 IN NS ns1.google.com.

;; ADDITIONAL SECTION: ns1.google.com. 158334 IN A 216.239.32.10 ns2.google.com. 158334 IN A 216.239.34.10 ns3.google.com. 158741 IN A 216.239.36.10 ns4.google.com. 158334 IN A 216.239.38.10

;; Query time: 68 msec ;; SERVER: ;; WHEN: Mon Sep 26 16:41:26 2011 ;; MSG SIZE rcvd: 310

mySQL FU

in one line, take a database, in stream replace content and stream into another db.

mysqldump original_db | sed ‘s/content_or_regex_to_replace/content_or_backref_replacement/g’ | mysql destination_db

Credit goes to d3in0s for his awesome forum post.

/System/Library/PrivateFrameworks/Apple80211.framework/Versions/Current/Resources/airport
Usage: airport <interface> <verb> <options>

    <interface>
    If an interface is not specified, airport will use the first AirPort interface on the system.

    <verb is one of the following:
    prefs   If specified with no key value pairs, displays a subset of AirPort preferences for
        the specified interface.

        Preferences may be configured using key=value syntax. Keys and possible values are specified below.
        Boolean settings may be configured using 'YES' and 'NO'.

        DisconnectOnLogout (Boolean)
        JoinMode (String)
            Automatic
            Preferred
            Ranked
            Recent
            Strongest
        JoinModeFallback (String)
            Prompt
            JoinOpen
            KeepLooking
            DoNothing
        RememberRecentNetworks (Boolean)
        RequireAdmin (Boolean)
        RequireAdminIBSS (Boolean)
        RequireAdminNetworkChange (Boolean)
        RequireAdminPowerToggle (Boolean)
        WoWEnabled (Boolean)

    logger  Monitor the driver's logging facility.

   sniff   If a channel number is specified, airportd will attempt to configure the interface
       to use that channel before it begins sniffing 802.11 frames. Captures files are saved to /tmp.
       Requires super user privileges.

   debug   Enable debug logging. A debug log setting may be enabled by prefixing it with a '+', and disabled
       by prefixing it with a '-'.

        AirPort Userland Debug Flags
            DriverDiscovery
            DriverEvent
            Info
            SystemConfiguration
            UserEvent
            PreferredNetworks
            AutoJoin
            IPC
            Scan
            802.1x
            Assoc
            Keychain
            RSNAuth
            WoW
            AllUserland - Enable/Disable all userland debug flags

        AirPort Driver Common Flags
            DriverInfo
            DriverError
            DriverWPA
            DriverScan
            AllDriver - Enable/Disable all driver debug flags

        AirPort Driver Vendor Flags
            VendorAssoc
            VendorConnection
            AllVendor - Enable/Disable all vendor debug flags

        AirPort Global Flags
            LogFile - Save all AirPort logs to /var/log/airport.log

<options> is one of the following:
    No options currently defined.

Examples:

Configuring preferences (requires admin privileges)
    sudo airport en1 prefs JoinMode=Preferred RememberRecentNetworks=NO RequireAdmin=YES

Sniffing on channel 1:
    airport en1 sniff 1


LEGACY COMMANDS:
Supported arguments:
 -c[<arg>] --channel=[<arg>]    Set arbitrary channel on the card
 -z        --disassociate       Disassociate from any network
 -I        --getinfo            Print current wireless status, e.g. signal info, BSSID, port type etc.
 -s[<arg>] --scan=[<arg>]       Perform a wireless broadcast scan.
                   Will perform a directed scan if the optional <arg> is provided
 -x        --xml                Print info as XML
 -P        --psk                Create PSK from specified pass phrase and SSID.
                   The following additional arguments must be specified with this command:
                                  --password=<arg>  Specify a WPA password
                                  --ssid=<arg>      Specify SSID when creating a PSK
 -h        --help               Show this help

Credit goes to d3in0s post showing true forum awesomeness.

/System/Library/PrivateFrameworks/Apple80211.framework/Versions/Current/Resources/airport -I
     agrCtlRSSI: -40
     agrExtRSSI: 0
    agrCtlNoise: -92
    agrExtNoise: 0
          state: running
        op mode: station 
     lastTxRate: 54
        maxRate: 54
lastAssocStatus: 0
    802.11 auth: open
      link auth: wpa2-psk
          BSSID: <removed>
           SSID: <removed>
            MCS: -1
        channel: 6
/System/Library/PrivateFrameworks/Apple80211.framework/Versions/Current/Resources/airport -s
                            SSID BSSID             RSSI CHANNEL HT CC SECURITY (auth/unicast/group)
                          <removed> <removed> -41  6       N  -- WPA(PSK/AES,TKIP/TKIP) WPA2(PSK/AES,TKIP/TKIP)

Doing a frame cap.

/System/Library/PrivateFrameworks/Apple80211.framework/Versions/Current/Resources/airport en1 sniff 6
Capturing 802.11 frames on en1.

You will see your airport icon changes to now hit ctrl+c to stop the cap

^CSession saved to /tmp/airportSniff813ZrA.cap.

curl -O http://python-distribute.org/distribute_setup.py
python3 distribute_setup.py

Enjoy!