/code - a blog

On Easily Replacing Text In Vim

Or more specifically: search and replace using the current visual selection.

Lately with my job, I've been spending more time refactoring old code than writing new code. Refactoring and manipulating code (and text) is something that I think Vim is great at, once you really understand the different modes and motions that Vim has, see Stack Overflow for a good summary. Vim's Search and Replace mode is what I've primarily been using and while refactoring, I found some Vim tricks I wanted to share.

Let's look at normal search and replace usage:

:%s/foo/bar/gc
Find each occurrence of 'foo', and replace it with 'bar'.

When dealing with complex functions or tricky variable names, I didn't want to type out my function name again into Vim's command line. I wanted a way to visually select my search text and just type my replacement text. I found some help on Stack Overflow again, finding out about the <C-R><C-W> shortcut to insert the current word under under cursor. The Vim Wikia explained this shortcut a bit further and I had enough ideas to write something out:

vmap <Leader>r "sy:%s/<C-R>"s/

So, visual map leader key + r to the command: yank the current visual selection to the s register, then go into command line mode, setup a string replace on the current buffer and use to load text from the s register. Then you can finish the replace command!

My final version include a tip from the Wikia article to run the search string through substitute() to replace newline characters, allowing for multi line/end of line searching. I had to switch to using the @ register format instead of the quote register format for vim to complain though.

vmap <Leader>r "sy:%s/<C-R>=substitute(@s,"\n",'\\n','g')<CR>/

To expand on this command, to find the number of occurrences of the visually selected word, before you run a replace on it:

vmap <Leader>c "sy:%s/<C-R>=substitute(@s,"\n",'\\n','g')<CR>//n<CR>

Other cool search aliases I found in vim:

• [I shows all instances of the word under the cursor in the document
• ]I shows all instances of the word under the cursor after your current location

One final tip, regex can be a dangerous tool kids, so always use the confirm flag when replacing!

Toggle OSX Audio Output From The Command Line

I spend alot of time at my computer and with that much usage, have both speakers and headphones for different hours of the day. I was initially physically swapping 3.5MM audio output plugs on my speakers, lotsa actual effort! I then figured out I could plug my headphones into the "rear" speaker output and switch output through the audio control in OSX, much easier! But that was even work, to open the audio panel and chose the correct source, so today I figured out how to toggle sources in Applescript and the command line. I then bound that script to a keypress and now I can toggle audio sources much easier!

Since I run a Hackintosh, I am using my motherboard's internal audio chipset with 8 channel output, although I'm assuming any Mac Pro or MacBook system with multiple audio output sources could use this. Internal speakers is regular out, Line Out is my headphones. Here is my audio panel:

At the beginning I found SoundSource and a few other menubar options for easily switching audio sources, but they still required work. I searched on Google and StackOverflow, but only found 30-50 line Applescript scripts which just controlled the Audio Panel, which seems klunky to me. The first piece of the puzzle was "switchaudio-osx" on Google Code which allowed me to control audio sources from the command line. Download the source, which contains a binary SwitchAudioSource.

Basic usage from a command line is to get a list of audio sources, shows your current audio source, and chose a new audio source:

ruin:~/$ SwitchAudioSource -a
Built-in Microphone (input)
Built-in Line Input (input)
Built-in Digital Input (input)
Built-in Output (output)
Built-in Line Output (output)
Built-in Line Output (output)
Built-in Digital Output (output)

ruin:~/$ SwitchAudioSource -c
Built-in Line Output
ruin:~/$ ./SwitchAudioSource -s "Built-in Line Output"
output audio device set to "Built-in Line Output"

That was enough to do what I wanted from the command line, but to control SwitchAudioSource from a keypress, I used Spark, an open source "shortcut manager", which I've written about previously. Spark allows to bind anything to custom keypresses. From here, it was figuring out enough Applescript to use SwitchAudioSource to toggle audio sources properly. Simple!

Code is here to copy, suit to fit your audio sources and and enjoy!

Dynamic Fabric Commands For Managing Cloud Servers

Fabric is a Python CLI tool for interacting with remote servers, that I've been pushing at work the last few months. It's great for organizing simple tasks to run locally or remotely, this blog is even being deployed using Fabric currently! It's a great tool.

The two main ideas of Fabric are that you have hosts and tasks you apply to those hosts. Fabric uses fabfile.py as its default instruction file, similar to a Rakefile or makefile. From a command line, you can call multiple tasks on a host and they will get applied in the order you called them. Hosts can be defined staticly in the fabfile through a variety of methods, but this gets hard once you have multiple hosts that can change easily with cloud computing/VPS solutions. I found a straight foward way to handle this was to have a task which defines which host I want to use, then the actual task I want to run on that host; calling fab production task1 or fab staging task1 to run task1 on my production or staging server.

This is great for 1-2 static servers, but becomes more work to manage when in a dynamic cloud based server environment. I wanted a better solution, but Fabric doesn't easily fit to this pattern. I found dynamic tasks had been attempted before before with Fabric and used that idea as a start. Here is what I came up with to create dynamic host tasks:

import sys
from fabric.api import *

### Before
def prod():
    """HOST: prod"""
    env.hosts = ['user@foo.com']
def staging():
    """HOST: staging"""
    env.hosts = ['user@bar.com']
### After
hosts = {
    'prod'    : ['user@foo.com'],
    'staging' : ['user@bar.com'],
}
def _createHost(hostName, hostList):
    moduleObj = sys.modules[name]
    def setHost():
        env.hosts = hostList
    #Doc string is readonly during creation?
    setHost.doc = 'HOST: %s ' % hostName
    setattr(moduleObj, hostName, setHost)
for host, hostList in hosts.iteritems():
    _createHost(host, hostList)
def test():
    run('ls')

So when Fabric runs, each host task is added as a function on the Fabric module and available to call.

Now my current blog server is hosted on Linode and a good portion of my work servers are on Linode as well. Linode has a decent JSON api, and Python library using that api, which I can use to used to get a list of all currently running Linode servers and their IPs. Extending my code above, you can get a dynamic list of your Linode servers in Fabric!

def _createHost(hostName, hostSize, hostList):
    moduleObj = sys.modules[__name__]
    def setHost():
        env.hosts = hostList
    #Doc string is readonly during creation?
    setHost.__doc__ = 'HOST: %sMB' % hostSize
    setattr(moduleObj, hostName, setHost)

def _generateLinodeHosts():
    sys.path.append('.')
    # Import api locally to hide from Fabric
    import api as linodeApi
    LINODE_KEY = "LINODE_KEY_GOES_HERE"
    linode = linodeApi.Api(LINODE_KEY)
    linodeList = [(x['LINODEID'],x['LABEL'], x['TOTALRAM']) for x in linode.linode_list()]
    for server in linodeList:
        ipList = linode.linode_ip_list(LinodeID=int(server[0]))
        publicIps = [x['IPADDRESS'] for x in ipList if x['ISPUBLIC']]
        _createHost(server[1], server[2], [publicIps[0]])
_generateLinodeHosts()
def test():
    run('ls')

You will want to add caching around your Linode calls, otherwise every usage of Fabric will generate a dozen api calls and be incredibly slow. I used pickle in my production setup, to save the Linode server list until I wanted to refresh it. This idea could easily be ported to Amazon EC2 or Slicehost or any other cloud provider, so happy hacking!

You can view all of this code on github here.

Shmoocon 2011

I attended my second Shmoocon earlier this month in Washington DC, a conference which I've begun to describe to my friends as a 'hacking and security' conference. Shmoocon is a wonderful mix of computer security folks, physical security folks (lockpicking), and hackers (folks just interested in how things work). It's fun to attend and find many examples that prove you really aren't as safe as you thought, you really can't trust most companies or people, but there is plenty that you can do about it, once you know to how to defend yourself. One of the best things Shmoocon offers is a different perspective on the world, which I always savor.

Recap

The weekend was quite busy: a drive down from Pittsburgh to my friends house just outside DC after work Thursday. I crashed hard that night and headed into town Friday morning. It was my goal to find some street food in DC and this taco truck on K ave (I think) provided the means! Food from trucks is so good!

Friday night, all Saturday, and Sunday were spent on a mix of attending talks, playing TF2 for Hack Fortress, and drinking copiously. There was an RIT alumni mixer Saturday night which showed my how many RIT alums are in DC!

New Ideas

Two talks I really enjoyed were "Computer Search and Seizure" and "Inside the App: All Your Data are Belong to Me." I only caught the tail end of the search and seizure talk, but it was being given by a lawyer from the EFF and seemed worthwhile enough to watch the talk again once it comes online.

The other talk covered sniffing iPhone backups for passwords and other "secure" info. Here is the summary:

Inside the App: All Your Data are Belong to Me
Sarah Edwards
Everyone knows their life is stored in their iPhones and iPads, but to what
extent? Forensic software can quickly and easily extract the data contained in
the default applications such as Address Book and Safari. This software can be
run by the most inexperienced of investigators and is often just a click of a
button to perform an “analysis”. iOS applications have become very prevalent
and many people do not know the amount of revealing data that can be found in
them. This presentation attempts to show the extent to which a person’s private
information can be exploited by doing simple analysis of iOS applications.
People have the right to know what data an app may store about them. This
private information may be used by malicious adversaries, intelligence agencies
or law enforcement. Take the time - know your apps.

This talk showed that many iPhone apps cache or store plain-text passwords and app data in an organized format that is easily accessible in the iPhone backup. One idea I grepped from this was using iPhone backups as a data export feature from websites that do not offer a better method. Several website/apps covered included LinkedIn, Facebook, and a few bank websites. Banks are thankfully good about allowing you to export data, but Facebook has had a mixed history, only in recent months allowing you a partial download of your data. Perhaps the iPhone backup might offer more data?

Overall, another busy and interesting weekend!

Upgrading To Blogofile 0.7

I saw recently on PyPI that blogofile, the Python blog engine that builds this blog, updated to version 0.7. This prompted me to upgrade my blog from 0.6 and detail the changes here, since the blogofile docs don't seem to have kept up. For the most part, this was discovered through trying to build my blog until it worked. I used the the blogofile init simple_blog and blogofile init blogofile.com projects as examples of "working" 0.7 projects and pulled code from them when my 0.6 code wasn't working.

Main Changes

• The blogofile main config has changed. This always seems to the change the most between versions, My recommendation is that it would be quicker to scrap your old config and just reconfigure everything. The main reason is that every config variable is now more OO, being attached to a blog object, cleaning up the mess of random variables previously:

-blog_auto_permalink_enabled = True
+blog.auto_permalink.enabled = True

• With the new config naming scheme, your templates must be updated. I did this manually since I had tweaked my templates and wanted to keep my changes. You must update all config references to use the blog object; the disqus config variable mostly:

-% if bf.config.disqus_enabled:
+% if bf.config.blog.disqus.enabled:

• Controllers have become more generic and the "blog" controller has moved to it's own subfolder/module. From what I understand with the development of blogofile, this has been done to point blogofile toward becoming more of a static file generating engine with a blog controller, not just a blog engine. This is a good thing, but requires rebuild most of your controllers directory. I would pull these from the simple_blog project.

• Likewise, filters for markdown and syntax highlighting have been updated, I would pull these from the simple_blog project.

See here for the diff of my main 0.6>0.7 commit.

New Stuff

With that out of the way, the cool things that 0.7 brings are:

• Support for drafts! Just add a draft: true to your blog header and you can keep your post in source control, but not have it published!

• Built-in linenums on source code now! Just add the linenums=true variable in your code tag.

1
2
3

$$code(lang=python,linenums=true) 
foo = 1
bar = 3

• Version increment! This version has been a long time coming, but I feel I am missing most of the "behind-the-scenes" code changes that have made blogofile more dynamic and are paving the way for more powerful stuff in the future.

Keep writing more code to make that future happen! To view the the source for this post and the rest of my blog, see my github project.