Tools & Utilities

Google Release Command Line Tool: GoogleCL

Having been raised on the good old fashioned ZX Spectrum and introduced to PCs via MS-DOS I have something of a nostalgic fascination with command line interfaces, there’s something beautifully simplistic about using a good command line – like you’re talking to the machine directly.  I’m not just talking about using the pseudo DOS-shell that comes with NT or the ubiquitous Unix/Linux command line, it’s the more exotic examples that pique my interest which is why I’m quite excited about the new Google command line tool.

Essentially GoogleCL is a Python application that can be executed at the command-line to make calls to various Google APIs, it currently offers limited support for Blogger, Calendar, Contacts, Docs, Picasa and YouTube but I’m certain that Google will deliver more features in the future.  In terms of security, there’s a one-time authentication process for each application whereby the command-line tool launches a page in your default web browser to grant access for the GoogleCL tool.  To me, the most interesting examples that Google provide are those allowing content creation…

There may not be many obvious ties to the world of Business Intelligence here as GoogleCL is still in its infancy but for now at least you could perhaps drive scheduling through Google Calendar, maintain distribution lists in Google Contacts or automatically upload reports to Google Docs.  I’m quite sure the possibilities will expand over time though especially since some major Google products are currently not included (e.g. Search, Gmail) – I, for one, will be watching with great expectations.

How To Install GoogleCL on Mac OSX Snow Leopard

If you’ve not heard the buzz already Google have released a Command Line tool called GoogleCL, you can install in in Windows by following Isaac Truett’s guide to”Setup GoogleCL on WinXP“ but if you’re using a Mac and you’d like to install it and have a play here’s a few simple instructions…

1. Enable your Root login (instructions from Apple in KBHT1528).
2. Log in as Administrator (bear in mind your normal user shouldn’t have Admin rights).
3. Download and install Xcode.
4. Download and install MacPorts.
5. Open up Terminal.
6. Edit your ‘paths’ file: sudo vi /etc/paths
7. Add a new line (press ‘i’ then scroll to the bottom first): “/opt/local/bin“.
8. Save the file (press ESC, then type “wq!”).
9. Close Terminal and re-open.
10. Type: sudo port install googlecl, and press Enter (this takes a while).
11. Log off as the Administrator.
12. Log back in as yourself and test (see examples).

FTP MGET – Get Multiple Files Without Prompt

Every now and again I have to use FTP (and Secure FTP) from the command line, in fact it’s actually my preferred method as it keeps my knowledge of the syntax nice and sharp rather than relying on GUI clients (though if you do need a good free FTP, SFTP and FTPS application you should try FileZilla). 

Today I had to retrieve a set of log files from a supplier, we can use the MGET command to fetch multiple files using wildcards (e.g. *.csv) but the default behaviour of MGET is to ask the user to confirm that they want to download every file – not very convenient if you’re talking about tens of thousands of logs!  Instead of resting a book and a penlid on the ‘Y’ key (I’ve seen it done) you can turn off the interactive prompts simply by issuing the PROMPT command to toggle on/off the prompts. 

Additionally if you’d like your FTP responses to be less wordy you can use the VERBOSE command which will pare down responses to the minimum.

Extract Rows from Files using QGrep

Whether you work with enterprise Data Warehouses, departmental Data Marts or live operational OLTP systems there’s a good chance that at some point you’ll have a need to import or export data using flat files.  Flat files are great to read because they’re typically in common easy to understand formats such as CSV, the problem comes when you’ve got extremely large volumes of data that exceed your ability to handle them in Excel (e.g. over 65,536 rows for Excel 2003) or when you absolutely must preserve the initial file format.

I had a requirement recently to extract all rows relating to just one customer from an input file containing tens of thousands of rows and I had to leave the structure intact so that the existing SSIS loader would recognise the file.  Sadly the standard Windows command-line utilities are a bit lacking in this regard but you can download the Windows Server 2003 Resource Kit which among other things includes the QGrep tool.

QGrep is simply a Windows equivalent of the Grep command available in Linux and Unix, there is no GUI but if you’re doing something simple the syntax is fairly straight-forward (see SS64.com for more info), in my case I needed to extract all records from “input.csv” containing the CustomerID “476226235076″ and place them in “output.csv” so the command was…

qgrep “476226235076″ inputs.csv > output.csv

Useful SQL Colouring Tool

I guess it depends on your background but if you’re anything like me you’ll prefer to have your SQL coloured and formatted, I began writing SQL in Microsoft SQL Server’s Query Analyser which colours comments green, keywords blue, aggregate functions pink and string values in red – and it’s something I (almost) can’t live without.

It’s become a fairly common scheme for viewing SQL and years of being used to colours mean that reading too much monochrome SQL drives me crazy so when I started this blog I was a concerned about how off-putting it would be if I posted reams of plain old SQL, luckily a bit of Googling led to me finding Instant SQL Formatter.

Instant SQL Formatter allows you to format and colour a SQL statement and then provides you with the HTML that you can paste right into your blog, there are also lots of options to tweak the formatting based on how you’d like to see the SQL laid out.  Here’s some example SQL I wrote using the tool…

– TESTING COLOURED SQL QUERY
SELECT Species,
COUNT(DISTINCT AnimalID) AS NumberOfAnimals
FROM tblAnimal
WHERE Species IN (‘Bonobo’,‘Vervet’,‘Chimpanzee’)
AND IsHealthy = 1
GROUP BY Species
HAVING COUNT (DISTINCT AnimalID) > 1

SQLBits V: Highlights from Friday Afternoon

Having attended the SQL Bits conference in Newport last month I decided to collate the little tips and tricks I learned at the event, in the challenging post-lunch session on Friday Thomas Kejser gave a talk on Designing I/O Systems for SQL Server – my notes from this talk are either short, sharp facts or references to thing I plan to research further…

The most important metrics are Throughput (measured in MB/sec or IOPS) and Latency (milliseconds).

A 10K RPM disk will give 100-130 IOPS, a 15K RPM disk will give 150-180 IOPS.

Performance can be increased by short-stroking disks, if you build a single partition on a drive using only 10% of it’s capacity this partition will reside physically on the outer edge of the disk.  Doing this reduces the amount of head movement required to read the data and improves performance.

Windows Server 2008 handles disk alignment automatically but for earlier versions the  offset should be 1024kb.

For testing the effect of configurations on performance use SQLIO.EXE or IOMeter, for stability testing use SQLIOSIM.

Check out the Pre-Deployment Best Practices Whitepaper from Microsoft.

The last talk of the day was by Allan Mitchell and was initially titled Common Integration Services Problems but was changed late-on to a more introductory Solving Problems in ETL using SSIS.  The only take-away I got from the talk was to check out a tool called BIDS Helper, I have no criticism of Allan though – he’s a good speaker but in this case I already understood the bulk of the content so it was hard to stay focused.