Ask the Relic

Large Graphs with Google Chart API

Yesterday, I saw forum post about predictions for when a video game (DNF) would get released. That prompted me to generate a multi-year vertical bar chart graphing the number of predictions per month/year over the last several years and I felt like sharing the process.

I love making graphs. I also love using Google's Chart API for making quick but decent looking graphs. They get the job done easily and since it is an HTTP API, it's easily to manipulate, share, and program. Unfortunately, the Chart API has a limit of 1000px wide or tall for your largest image, which I easily hit with my multi-year graph, but you can get around that by using the Python Image Library (PIL) to stitch together multiple smaller graphs.

While making this graph, I identified three main points that would probably help in making future graphs:

1. Normalize your input data. Get a uniform date format that is easy and flexible to work with first, separate from your graph making code.
2. Build a blank graph first, get the layout/look down, then build the graph with data.
3. Know when stop using the API or your code to build the graph and switch to Photoshop to finish off the details. Time/effort trade-off, but for me, final layout of the graph and fonts, text, minor things are way easier to finish outside of the code.

On to the code! I'm skipping the data formatting/normalizing parts from this code, it's mundane. This is using pygooglechart 3.0 and PIL 1.6, Python 2.6. The full code is located on Github.

First, we setup out basic graph. With the 1000px limit, we can create three years of graph at a time.

Python's list comprehensions make generating random data formats for graphs quite easy! Here we are using the calendar modules built in listing of month abbreviations to generate the months, then the year date below.

	#build the monthly abbrevations
	months += [x for x in calendar.month_abbr if x]
	#build the empty year array
	years += [''] * 12
	#then fill in January position
	years[y * 12] = ' %s' % start_year

After we have generated multiple three year graphs, we can join them together into one huge graph using the PIL to crop and paste them together. We create a large blank image first, then paste the first graph, then crop the other future graphs to remove their y axis labels. The specific crop values come from trial-error, but they are easy to figure out.

	#new large canvas to work with
	nim = Image.new("RGB", (5000,300), "white")
	for file in glob.iglob('20*-*.png'):
	    #if the first graph, start at 0
	    if counter == 0:
	        im = Image.open(file)
	        nim.paste(im, (0,40))
	    #otherwise, crop the y labels from future graphs
	    else:
	        im = Image.open(file)
	        im = im.crop((18,0,825,250))
	        nim.paste(im, (x,40))
	        x += 790

Then we wind up with a huge 5000px x 250px graph here. Having a blank yearly based template is useful for alot of data, but this could be generalized easily to daily data, or hourly data.

Accessing Google Reader via OAuth and Python

Last month, Google Reader announced support for accessing user data via OAuth. Previously, access was unofficially allowed using the ClientLogin method, which required the user's login and password. OAuth seems to be the recommended access method for the future, due to security it provides for a user. I've finally had a chance to figure out OAuth using Python and how to get your Reader data, so I wanted to share my method.

For an example of the ClientLogin method in Python, take a look at my libgreader library on github. It automates the ClientLogin auth and tokens that you are required to have for any requests. I'm working to merge this code below into a more generic Google Reader library supporting both auth methods.

I've written my code using v1.1.3 of the OAuth2 python library to help wrangle OAuth. This was my first foray into OAuth and using the library certainly helped for figuring out all the required data and security that you need to have on every request. If you haven't used OAuth before, definitely check out http://oauth.net/ and the Google docs for using their OAuth implementation. If you just want to experiment with the process not using Python, use this interactive JS/AJAX app.

The Code

First, let's setup some imports and definitions. You can also view this source code on Github.

	import urlparse
	import oauth2 as oauth
	
	#Unsecured http for testing, should be https for production
	scope = "http://www.google.com/reader/api"
	#Alphabetical list of a user's subscriptions
	sub_url = "%s/0/subscription/list" % scope
	#JSON list of a user's unread feed items
	reading_url = '%s/0/stream/contents/user/-/state/com.google/reading-list' % scope
	
	#Google Auth urls
	request_token_url = "https://www.google.com/accounts/OAuthGetRequestToken?scope=%s" % scope
	authorize_url = 'https://www.google.com/accounts/OAuthAuthorizeToken'
	access_token_url = 'https://www.google.com/accounts/OAuthGetAccessToken'
	

Now, you have to setup a consumer key with Google, for the future domain your webapp will run at. The url to manage your domains is https://www.google.com/accounts/ManageDomains

	oauth_key = "www.asktherelic.com"
	oauth_secret = "XXXXXXXXXXXXXXXX"
	
	consumer = oauth.Consumer(oauth_key, oauth_secret)
	client = oauth.Client(consumer)

From here, most of the code mirrors the example code from the OAuth2 library, using a 3-step process to verify and get your required tokens from Google. In step 2, you have to manually open the link and authorize your script with Google.

	# Step 1: Get a request token. This is a temporary token that is used for 
	# having the user authorize an access token and to sign the request to obtain 
	# said access token.
	
	resp, content = client.request(request_token_url, "GET")
	request_token = dict(urlparse.parse_qsl(content))
	
	print "Request Token:"
	print "    - oauth_token        = %s" % request_token['oauth_token']
	print "    - oauth_token_secret = %s" % request_token['oauth_token_secret']
	print
	
	# Step 2: Redirect to the provider. Since this is a CLI script we do not
	# redirect. In a web application you would redirect the user to the URL
	# below.
	
	print "Go to the following link in your browser:"
	print "%s?oauth_token=%s" % (authorize_url, request_token['oauth_token'])
	print
	
	accepted = 'n'
	while accepted.lower() == 'n':
	    accepted = raw_input('Have you authorized me? (y/n) ')
	
	# Step 3: Once the consumer has redirected the user back to the oauth_callback
	# URL you can request the access token the user has approved. You use the
	# request token to sign this request. After this is done you throw away the
	# request token and use the access token returned. You should store this
	# access token somewhere safe, like a database, for future use.
	token = oauth.Token(request_token['oauth_token'], request_token['oauth_token_secret'])
	client = oauth.Client(consumer, token)
	
	resp, content = client.request(access_token_url, "POST")
	access_token = dict(urlparse.parse_qsl(content))
	
	print "Access Token:"
	print "    - oauth_token        = %s" % access_token['oauth_token']
	print "    - oauth_token_secret = %s" % access_token['oauth_token_secret']
	print
	print "You may now access protected resources using the access tokens above."

An annoying process, but once completed, you can create an authorized client which can hit any Google Reader url and access data. First, let's start off with the user's subscription list. Then the user's reading list.

	#Authorized client using access tokens
	token = oauth.Token(access_token['oauth_token'], access_token['oauth_token_secret'])
	client = oauth.Client(consumer, token)
	
	#Get user's subscription list
	resp, content = client.request(sub_url, 'GET')
	print content
	print 
	
	#Get user's reading list
	resp, content = client.request(reading_url, 'GET')
	print content

In this CLI example, the user experience is pretty disjointed, but in a webapp setting, you would like for a more fluid experience. The way to do that is using a callback: a location that google will punt the user to after the 2nd step, when they have authorized your application. Checkout the code below for a simple example or the google docs for options.

	#Google Auth using a callback
	callback = "http://www.asktherelic.com/thenextstep/"
	request_token_url = "https://www.google.com/accounts/OAuthGetRequestToken?scope=%s&oauth_callback=%s" % (scope, callback)

Possible Issues

The Google Reader API is still pretty undocumented, most knowledge comes from sniffing Google Reader traffic that their webapp generates or experimenting on your own. One difference between the ClientLogin and the OAuth login is that ClientLogin method requires your script to get a token from the Reader API. I haven't seen the OAuth method require that token in my testing, but it might be something that is still required.

Another thing I noticed while testing, with the 2nd OAuth auth step that occurs when you have multiple Google accounts, is that Google Apps accounts don't seem to work with my code. I'm using a regular gmail account but Google Apps appear to be a different affair.

Goodluck with the Reader API.

Awesome OSX Software

Many years ago, when I switched to OSX from Windows, I quickly knew it was the right choice to switch. As the years have passed and my knowledge of all three major platforms(OSX, Windows, and Linux) has increased, I've become more confident and happy with my choice of OSX. While the OS defines how you interact with the software, but the software defines what you can accomplish and enables you to do so. Here is a selection of awesome software which I've found really helps me out. You can also check out my last post about Terminal and the hacks I use here.

Spark

Spark is a shortcut manager that has been around for years, but I just recently found out about. I am a very keyboard driven user who loves to customize things and Spark lets me do that. Spark allow you to rebind keyboard shortcuts to do pretty much anything: Applescript, launch Applications, open Documents, call window/menu commands in specific programs or call other keyboard shortcuts. It's very versatile and has replaced a few other applications for me.

Here is my current Spark configuration, which I am mainly using to reconfigure iTunes. I listen to iTunes 24x7 and like to be able to easily rate the currently playing track. I have CTRL + F15-F19 keys setup to do so. I also have a CTRL key setup to display the current track, along with album cover, current position in the song, and song length. Having all of this info is really nice, because I have seen other pay-for iTunes control programs skimp and not provide it.

One of the things not shown, is that you can bind keys per-application or system wide. I have a shortcut in iTunes for Apple + D to bring focus to the search music window because the default shortcut key is Apple + Shift + F, which is awkward for a shortcut I use frequently. iTunes has Apple + F bound to set the visualizer to fullscreen mode, which I never even use the damn visualizer.

Some of this might seem minor, but the overall package that Spark provides is stellar. You can export and import your database of commands, which is great for future portability and backups. The Spark Daemon seems to average around 10MB of memory usage, which is less than other iTunes control programs I've seen and it does more than other programs. Combined with the expandability to control any other program, I think Spark a good choice.

iStat Menus

As you have seen, I love being able to customize and tweak my system. iStat Menus is one of the best packages I have seen to monitor your system and what it is doing. It is very much a power user program, most people don't need to know as much info as it provides, but you do want that info, iStat provides it very well. The website for iStat provides a very thorough description, worth checking out.

Teleport

Teleport is another piece of software that I just found out about recently, but has been around for awhile: it allows you to share control of 2(or more) OSX systems via 1 keyboard/mouse. Very similar to Synergy, which allows you to share control between any OSX/Windows/Linux system, but I always found Synergy to be buggy and not that fluid. Teleport works quite well, you can configure it to switch systems when you get to the edge of the screen and it works near instantaneously. If you have a main machine and a laptop, you can have Teleport run in the background on both and control the laptop via the main system, whenever the laptop is plugged in. Teleport is nearly invisible and is one of the reasons it is so good.

Sequel Pro

As a developer, I work with mySQL frequently and installing phpMyAdmin is annoying. Sequel Pro is a mySQL database management program, which has evolved quite a bit over the last few years and is way better than the default mySQL admin tools, which have not seen an update in years.

Sequel Pro lets you create/edit/delete tables using the OSX Cocoa goodness. It has built-in support for SSH tunnels, so you can securely connect to your mySQL server on your development/production server and view/edit data there. You can import/export data as well. Sometimes working in a console isn't always the best choice, especially when dealing with lots of data. This program is a great option beyond the console.

These are great OSX programs that I use all the time. Have any other programs that you like? Post them in the comments!

My Perfect OSX Terminal Setup

As a programmer and a part-time sysadmin, I spend a huge amount of time in the OSX Terminal and find it be one of the better CLI environments I've used, after some tweaking. Windows + Powershell or CMD (shudder) is just terrible. Linux with Xterm comes close, but doesn't have the same easy usability that I enjoy about the OSX Terminal. Today, I'd like to share my tweaks and explain what I enjoy about them.

Here is my general color scheme/terminal look. I'm running OSX 10.6.3 with 64-bit Terminal. All of these tweaks depend on SIMBL, a bundle loader which loads these tweaks. I'm running version 0.9.7a.

Visor

Visor is a "system-wide terminal on a hot-key". I have Visor bound to Apple ~. Having simple access to the Terminal whenever I want is very important.

Visor also adds "copy on select", a very useful little tweak for text. Additionally, it allows you bind the Terminal to a specific position; having it drop down from the top and take up 50% of the screen, reminiscent of the Quake style visor. I bind my Visor to use the full screen.

Terminal Tab Switching

Visor only uses one main window, which means you can have multiple tabs in the window. Getting easy access to the specific tab I want is very important. Chrome, Firefox, and many other tab based applications bind Apple 1-9 to the specified tab location and I like having that functionality in Terminal. I rarely work with more than 9 tabs anyways.

TerminalTabSwitching was originally posted as a 32-bit plugin for SIMBL 3 years ago here. Since then, Terminal and OSX went 64-bit and the plugin stopped working. dabeeeenster has ported the plugin and you can download the latest version on github.

Another useful tweak I do is bind the left and right bracket, Apple [, to select the next or previous tab, like in every other program. Terminal defaults to using curly brackets, Apple {, to switch tabs, but doesn't let you change that. In the OSX System Preferences, under the Keyboard settings, you can rebind window commands to any key you want and I do so!

Terminal Colors

This is mostly cosmetic, but when in Vim and coding, having the specific color for a function or variable is useful and every color has huge implied meaning. Being able to better select the right color and using more eye pleasing colors is useful.

TerminalColors is a plugin that updates the default colors to be less jarring and lets you use the standard OSX color picker to choose any color you want. Downloadable here.

Addendum: If you are getting "Terminal version errors" or something equivalent, you may have to modify your plugin to support the latest version of Terminal. Goto ~/Library/Application Support/SIMBL, right click on the broken plugin and select "Show Package Contents". Then open the Contents folder and open the Info.plist file with a text editor. Generally it is the MaxBundleVersion number that gives problems. Increase that number until it works, I'm using 280 for my bundles and Terminal is at version 273 as of now.

These tweaks help me use Terminal better and make my life easier. If you want more tips on OSX Finder and Terminal integration, check out my other post here. Hope they help you as well!

Fixing MacFusion with OSX 10.6.3

MacFusion is a great little app that allows you to mount network locations over SSH, which I've mentioned before. With the latest 10.6.3 update in OSX, the latest version MacFusion 2.0.3 breaks. Unfortunately, the developers of MacFusion haven't touched the app in over 2 years.

I found a fix on the MacFusion Google Group. A third-party developer, nall, fixed the problem and updated the binary to 2.0.4, available at http://github.com/downloads/nall/MacFusion2/Macfusion-2.0.4-SL.zip. This version works fine for me on 10.6.3. While this might not be a long term fix, it works now and I will update this post if an official way comes out.

This latest version also now correctly shows the volume size for the mount point, which previously would only show "1TB" for me. Hooray for working software!