Following on from Linux – Generating file manifests and then checking them I was always getting the same questions …

How long left on the manifest Buzz ? How long left on the verification Buzz ?

And I HATE having to turn around an say … I don’t know …

The problem with the usual command line method is that it give no indication of progress, and by extension no indication that it was infact running and not ‘hung’ …

As such I have now added the ‘manifest’ command set to the Sysadmin toolset

The manifest command take two data types, the first is a folder path from which to build the file manifest from, the manifest itself is also compatible with the “md5sum –check” function.

[FLOWPLAYER=https://content.screencast.com/users/D.Busby/folders/Jing/media/dd41bc92-17d3-4c34-8d46-80188f13aff2/00000337.mp4,900,300]

The second is the path to the manifest itself, in this case the manifest command will verify each file against it’s entry in the manifest:

[FLOWPLAYER=https://content.screencast.com/users/D.Busby/folders/Jing/media/2d3eb196-87ef-4464-982b-5b9481c000fe/00000338.mp4,900,300]

At each point the command give you an indication of it’s current status, however this does come at a small cost, the script has no concept of the size of you console and as such will always render out the same number of character meaning if you console is not wide enough it will not render correctly, in the videos I have the console on a high resolution monitor as can be seen each video itself is 900 pixels wide.

This process is CPU intensive (20-45% on one core of a intel core2duo 2.8GHZ) and uses around 140KB of memory.

I had a major issue facing the iconv functionality of the sysadmin toolset namely due to rushed coding.

When loading a file to be re-encoded the entire file was loaded into the buffer, encoded as whole and written out to the new file, this of course meant the  memory usage was roughly double the size of the file to be converted plus any overheads to do with the encoding itself.

Today I had need to convert a 1.3GB sql file, needles to say the script was crashing out with a memory error.

As such I have now completely re-written the function it now processes the file in 1kb ‘chunks’, moving the load to the CPU, this process is now very cpu intensive the the memory overhead is minimal (during test processed the 1.3GB file using 113kb of memory!!!).

[FLOWPLAYER=https://blog.oneiroi.co.uk/uploads/2009/09/sysadmin-iconv.mp4,487,417]

Also I have now added BOM (Byte order mark) detection:

[FLOWPLAYER=https://blog.oneiroi.co.uk/uploads/2009/09/sysadmin_oconv_bom.mp4,515,473]

laurent gaffie has produced a proof of concept remote BSOD affecting windows vista /7.

It is advised at this time to block all NETBIOS and SMB trafic on your network as there is currently no patch available.

Read the entry here: https://seclists.org/fulldisclosure/2009/Sep/0039.html

At the time of writing this entry I tested this on a Windows Vista VM (fully patched).

MS - Security Advisory

This issue has come about whilst having to migrate a positively huge number of files, and have to check the integrity of the transfer.

Build the manifest

find /path/to/folder -type f -print0 | xargs --null md5sum > /path/to/manifest

• -type f : This flag tells find to only return files
• -print0: This flag tells find to null terminate strings, this allows us to take files with spaces
• –null: This flag tells xargs to accept null terminated strings
• NOTE: PUT THE MANIFEST OUTSIDE THE FOLDER YOU ARE INDEXING!

Checking the manifest

md5sum --check /path/to/manifest | grep FAILED

The above will return all failed checks, if you want a simple count (maybe for automated reporting) just add | wc -l

FAQ

How big is the manifest?

This depends entirely on the length of your filepaths, taking UTF-8 as an encoding example each char is 8bits or 1byte, each manifest line consists of the md5hash, a space and the filepath as the filepath length varies there is no exact way to estimate the filesize of the manifest.

However each line is always 32 + 1 + len(path) bytes.

The more sub directories you have the larger the manifest size will be.

How long does the manifest take to build?

This depends on the number of files you have to index, along with any other factors such as network shares, in test runs 2819 files indexed in 1.493 seconds.

Finaly it is ready and in a state I am happy to release it.

Features

• Complete code re-write, much more efficient and easier to update
• Flowplayer 3.1.1
• License detection, will only use the commercial version if you enter a license!
• Playlists
• Better wordpress API integration, all settings now stored in wordpress no more config files!
• Same admin interface

(Videos from Apple.com)

[FLOWPLAYER=https://movies.apple.com/movies/fox/avatar/avatar2009aug0820a-tsr_h.480.mov|https://movies.apple.com/movies/independent/thehorseboy/thehorseboy_h.480.mov,480,205]