One Man Clapping

Running Multiple Cassandra Nodes on a Single Host

2010-03-11T13:00:00.007-07:00

One of the first Cassandra tickets I worked on had me reviewing some code that visualized the node ring. Properly testing the code required that I run a cluster.

But I didn't have access to a cluster. Neither did I feel like creating a virtual cluster by building a VM and cloning it several times. What I wanted was to run several instances of Cassandra on a single machine with multiple interfaces, all pointed at the same compiled code (without multiple svn checkouts).

The Cassandra wiki explains how to tweak Cassandra settings by editing cassandra.in.sh, but doesn't explain what needs to be done to run concurrent instances.

It turned out not to be too difficult. I figured it might be daunting enough to Cassandra noobs (of whom we're seeing more of lately due to some great exposure), that a blog post might be helpful.

This tutorial assumes that you'll want to run multiple instances of Cassandra on code built by ant and not a standalone jar. I am also assuming that you are a) just playing around, or b) intend to do some development. This is not a tutorial explaining how Cassandra should be run in production.

Note: I apologize for the way this looks. Blogger is not a friend of ordered lists.

Make sure you've got aliases to localhost (e.g.: 127.0.0.2, 127.0.0.3, etc.). Mac OS X doesn't have this enabled by default, so you'll have to manually create aliases:
```
sudo ifconfig lo0 alias 127.0.0.2 up
```
```
sudo ifconfig lo0 alias 127.0.0.3 up
```
Decide where you're going to keep things. You can keep them with your code, but that just isn't neat. Pick a directory somewhere, call it $cass_stuff.
Then, for each node in your little cluster, do this:
1. From your svn checkout, copy the conf directory into $cass_stuff. You can rename it to something like conf0 (or conf1, etc.). I'll assume $conf from here on out.
2. Copy bin/cassandra.in.sh to $cass_stuff. Give it a name that helps you associate it with the conf directory you just created (node0.in.sh or whatever).
3. Open node0.in.sh in an editor and make the following changes:
  1. Hardcode cassandra_home to the location of your trunk. This will give you the flexibility to run Cassandra from anywhere.
  2. Set CASSANDRA_CONF to the conf directory you just created.
  3. In the JVM_OPTS change the jdwp address= setting. The default is 8888, but you should include the unique IP you chose for this node along with the port, e.g.: 127.0.0.2:8888. Not specifying a host causes the debugger to bind to 0.0.0.0:8888 and you'll have port binding problems when you bring up more than one node.
  4. pick a unique port for com.sun.management.jmxremote.port, but make sure you have at least one node listening on 8080 since all the Cassandra tools assume JMX is listening there. Unfortunately, you can't pick the JMX host, 0.0.0.0 is assumed. I was under the impression this could be changed by specifying java.rmi.server.hostname, but had no luck going down that road. (Please leave a comment if you figure out a way for this to work, but I think it might be hopeless.)
4. Open $cass_stuff/$conf/storage-conf.xml in an editor and make the following changes:
  1. specify unique locations for CommitLogDirectory and DataFileDirectory. Don't bother with CalloutLocation or StagingFileDirectory.
  2. replace ListenAddress with the IP of your host.
  3. replace RPCAddress with the IP of your host.

To run you may wish to use another script for each node:

#!/bin/sh

CASSANDRA_INCLUDE=$cass_stuff/
export CASSANDRA_INCLUDE
cd
bin/cassandra -f

One downside to this approach is that if you're tracking trunk, it is your responsibility to make sure you notice changes to the default storage-conf.xml and cassandra.in.sh and apply them to your environments.

Cassandra is supported by an active and welcoming community. If you'd like to learn more about the project, check out our wiki, mailing list or hop on #cassandra on freenode.

Dear Entrepreneurs, this is something I would pay for...

2009-12-15T18:00:00.004-07:00

Dear Entrepreneurs,

This is something I would gladly pay $20 a month for...

A device that, according to my tastes, downloads new music from the Internet whenever it connects. I would be able to listen to music without restriction while I am disconnected from the network. I wouldn't own the music, except for roughly 20 tracks a month that I select which would then become mine as MP3s (for FLAC or whatever DRM-less technology makes sense). I could then load them into iTunes, give them to my brother, or (if I'm feeling sinister) make them available on a P2P network.

The music could come from anywhere: iTunes, Amazon, The Labels, or artists themselves.

The content sources exist. The recommendation engines exist. Devices exist.

I suspect the audience/market exists. (At least, I hope so. If not, and nobody is willing to pay for music, we're going to need to find another model. And it will still necessarily involve a money exchange between producers and cosumers and/or advertisers.)

Is there such a system already?

Christmas Mix 2009

2009-12-13T22:00:00.000-07:00

I've been making Christmas mixes for my family the last couple years. It's not your typical Bing Crosby stuff, and requires some digging on my part. I finally started blogging about it last year and think I'm going to make it a tradition. So here goes... Christmas with an indie slant. And I did a better job checking on the lyrics this year for family appropriateness.

The links this year are coming at you from Lala by way of Google. Message me if things stop working. (This blog post has turned out much like my Christmas shopping: it gets sloppy towards the end.)

1. "Holiday Road" by Matt Pond PA. This is the only repeat from last years list. I love this song because the vacation movies still connect with me at a level I am entirely uncomfortable with.

2. "Blue Christmas" by Dread Zeppelin. Believe it or not, there is a nice smattering of Christmas to choose from with these guys. Where else can you get Elvis, Led Zeppelin, Reggae and Christmas in one track?

3. "Christmas is Going to the Dogs" by Eels. Hard choice between this and "Everything's Gonna Be Cool This Christmas".

4. "I Wish It Was Christmas Today" by Julian Casablancas. This one is for the kids. I wish that I could still feel the way I did when I was a young boy after Thanksgiving. Christmas, although only four weeks away, seemed like it sat on the other side of eternity. As an adult, it comes and goes so fast I barely have time to enjoy it. Message to kids: enjoy it while you can. Responsibility steals the fun from Christmas!

5. "Christmas Time is Here Again (Bring Out the Joy!)" by My Morning Jacket. Peaceful. I'll let you google for this one. It's a live take from a radio broadcast.

6. "Listening to Otis Redding at Home During Christmas" by Okkervil River. Not a traditional Christmas tune, but a good one to follow MMJ, if only for the indie vibe. This song reminds me of "New Slang" by The Shins, but with less jade and desperation. Slightly more hopeful. :)

7. "X-Mas Card" by MU330. Not my normal thing, but the instrumental intro with the horns is fun.

8. "Yule Shoot Your Eye Out" by Fall Out Boy. If you haven't checked out "Can You See Santa From the Southside," now is the time to skedaddle over to Amazon and do so.

9. "Baby, It's Cold Outside (Mulato Beat Remix)" by Louis Armstrong and Velma Middleton. Shopko gave away a Christmas sampler in 2004 and this was on it. This is, by far, the Christmas album that gets play in our house (not the one this song links to). It comes on while we're preparing meals and we find ourselves breaking frequently to get our grooves on. No kidding. Six people from 2 to 35 shaking a leg in the kitchen.

10. "O Come All Ye Faithful" by Weezer. Traditional Christmas tune done right by a modern band.

And some bonus songs from last years mix:

Bonus 1: "Fairytale of New York" by the Pogues. This one is definitely not for the kids and is a guilty pleasure of mine. Who can resist: "You're a bum, you're a punk / You're an old slut on junk." Ahh, the holidays.

Bonus 2: "Frosty the Snowman" by the Cocteau Twins. Year after year, my favorite Frosty rendition.

Building Cassandra Thrift Bindings on OS X

2009-11-02T17:00:00.001-07:00

A few weeks ago, I came to Rackspace to work full-time on Cassandra in their cloud division. So far, I'm having fun and learning new things. Apart from Cassandra, one of the projects I get to figure out is Thrift. Thrift is a tool that allows you to define a service interface and then generate stubbed service bindings in different programming languages. A programmer then takes the generated code and makes it do the things it is supposed to. In an ideal world, this simplifies the process of, say, stubbing in a PHP client that can speak to a server stubbed in Java.

Right now, I'm the lone wolf in the office doing development on OS X. The glitches so far have been minor, but I was forewarned that I might want to reconsider [using linux] when it came time to work with Thrift. Well, that time started today. I've been a faithful linux user for about 10 years, but I've been a faithful Mac user even longer. I'm not ready to make the switch to full-time linux yet; I like my Mac.

Fortunately, Google was my friend when it came to figuring out the secrets of building Thrift on OS X. Credit goes to Nathan Ostgard and his blog post for getting me going in the right direction.

1. You definitely want to install macports.
2. Install boost and log4j
sudo port install boost
sudo port install jakarta-log4j

3. Download and install thrift
curl -o thrift.tgz "http://gitweb.thrift-rpc.org/?p=thrift.git;a=snapshot;h=HEAD;sf=tgz"
tar -xvf thrift.tgz
cd thrift
echo "thrift.extra.cpath = /opt/local/share/java/jakarta-log4j.jar" > ~/.thrift-build.properties
./bootstrap.sh
./configure --prefix=/opt/local

4. You're going to get an error during configure:
./configure: line 16440: syntax error near unexpected token `MONO,'
./configure: line 16440: ` PKG_CHECK_MODULES(MONO, mono >= 2.0.0, net_3_5=yes, net_3_5=no)'

I couldn't figure out how to tell configure "no csharp, please" through the command line, so I just commented out lines 16439-16442 and ran configure again:

./configure --prefix=/opt/local

5. You know the drill:
make
sudo make install

That's it for Thrift. The next step is to generate the Cassandra client. The Cassandra wiki has steps to generate a python client. This works fine except that the thrift python module was installed to a place where the OS X python can't see it. You'll get the following error if you try to run Cassandra-remote:

Traceback (most recent call last):
File "./Cassandra-remote", line 11, in
from thrift.transport import TTransport
ImportError: No module named thrift.transport

There is probably a right way to fix this problem, a way that is right for OS X, but I had no patience. I added the the following line to my ~/.profile:

export PYTHONPATH=/usr/lib/python2.6/site-packages

Restart terminal, navigate back to the directory where the python client was generated and try again. Cassandra-remote should spew out a verbose usage directive.

That's it; you're done.

If you found this useful, or have feedback, please let me know. I use gmail (gdusbabek). I also emit the occasional tweet; just follow gdusbabek.

If you're interested in learning more about Cassandra, there is an active and helpful IRC channel (#cassandra) on freenode and mailing lists as well. The wiki also contains useful information for beginners.

id3 for Python

2009-09-14T22:00:00.068-06:00

I've been meaning to package up some python code I wrote earlier this year and release it for free as open source software. Several things held me back from doing this. The biggest reason, by far, is that I'm still not proficient at python, and feel like I'm exposing myself by putting this code out for the world to see.

But then I remembered that my blog doesn't have a lot of readers anyway. So no worries there. And besides, maybe I can garner some constructive criticism to make my python code better. :)

http://www.dusbabek.org/~garyd/id3_python/

This library is capable of reading most any correct Id3v2.3 or 2.4 tag, some incorrect ones, and then fails gracefully when things get hopeless. (I run smoke tests on my mp3 collection, which has a lot of nasty debris from the Napster years.)

It supports unicode, and does a good job handing PIC/APIC tags. I should also mention it is in production at Tagfriendly.

For those that do come across this, I'm still trying to figure something out. All the documentation I've come across says that I should structure my directories as such:

MyModule/
    id3/
__init__.py
        stuff.py
    setup.py
    tests.py

I include id3.py inside the id3/ directory because that's where I think it should go. But then when I build and test the module, the only way I can access the code is if I import id3.id3, but I wish I only had to import id3. Clearly, I'm not doing something right.

My solution, and I know this isn't right, is to do away with the id3/ directory altogether and just have id3.py rubbing elbows with setup.py and test.py. Anyone know what gives?

Well, enough of all that... id3 for Python is available at http://www.dusbabek.org/~garyd/id3_python/.

P.S. Thanks to my friends on IRC for reminding me about this.

My New Workout

2009-08-31T18:00:00.002-06:00

I have been attending the gym regularly, religiously even, for the last 10 years for early morning workouts. What started out as short 20 minute workouts for an out-of-shape 25 year old have turned into 45 minute cardio sessions where I routinely burn 900 calories. It's been a very good thing for me; I wouldn't trade it for the world.

When I started, I would bring headphones and watch the morning news. Then began a succession of portable music players that started with a Diamond Rio 500, which I cherished, and culminated with the two iPods I currently use. Over the last several years, gradually, the monotony of a daily workout began combining with my music against me, hampering my motivation. Maybe I can chalk it up to age, or plateauing, but regardless of the cause: I needed to find something else.

I thought that an iPod video (30 GB) would help, but it turns out the screen is much to small to watch while the rest of my body is moving on a treadmill or elliptical machine. (It is still great on an airplane though.) I've been using an iPod shuffle for the last two years, alternating between general conference sessions and music. I tried podcasts for a while, but the iPod has such a crappy podcast interface--you can't queue them up in a playlist, and I don't want to fiddle with buttons in the middle of my workout.

Total frustration.

Then something happened. My gym began installing televisions on all the cardio equipment. I thought my problem was solved--I'd be able to go back to just bringing a pair of headphones again. That joy was short lived, as I realized that most of what's on television in the early morning is generally crap (informercials) and sometimes downright crude (infomercials for porn--no kidding!). And the news is, well, frankly: not worth watching anymore. Back to square one...

I don't know why it took so long for the idea to sink in, but I realized that I could connect my iPod video directly to the televisions on the cardio machines using a $5 composite cable I already owned (thank you, Sony), and watch my iPod videos on the TVs. I finally gave it a try this morning. I watched two TED talks, and an APM Marketplace podcast about high-frequency trading. The best part is that when it was all over, my 45 minute workout felt like only 15 minutes had gone by!

Undeniable WIN!

My next task is to find more podcasts that don't suck. I've already found a decent language training podcast, but what would be really nice is to get a hold of some of the Google tech talks, as they are usually excellent and I don't mind watching them over (indeed, some of them ought to be watched multiple times to absorb the information). I couldn't find them through the iTunes store, but maybe some kind soul has been kind enough to make them available on the outside. Google, ironically, has not. In fact, the Google tech talks are strewn across video.google.com and youtube.com now that Youtube is part of Google. The older video.google videos are easily downloadable, but the Youtube ones are not. At least, not without some real work.

Then again, there are enough [free] things at iTunes U to keep my mornings occupied for a long time. They have a special section just for computer science educational content. Double plus woot!

Anyway, all this is a long way of saying something short: iPod + cardio TV == excellent workout.

Adventures in Javascript

2009-07-22T08:00:00.004-06:00

Note: Blogger doesn't give me a good way to preview the image before it goes live. If it is too small, my apologies. I'll try to get that fixed before your news reader pulls the feed.

The Question
Sometimes I think programmers tend to accept things the way they are without really questioning them, especially when it comes to language quirks. This is something I remember coming across when I was first learning Javascript. I was reminded of it recently while reading a Javascript book.

Javascript pros will probably quickly recognize what's going on here.

At first I thought the third definition of MyFunc would wipe out the others. And just to verify that declaring a function in that manner normally makes it into the assumed contexts, I did it with YourFunc.

What gives?

Go ahead, take a guess...

The Answer
The interpreter evaluates standalone functions before the other expressions. So it is as if 'function MyFunc...' were written before everything else. This can be verified by calling MyFunc() in the first line of the script. The expressions (which include assignments) are then evaluated. So what appears as the second assignment of MyFunc is really the third, and the third is really first.

My Take on Newspapers in America

2009-07-21T18:00:00.009-06:00

I have read numerous articles and blog posts over the last 12 months about the decline of the newspaper industry. Having spent a few years in the trenches myself, and having opinions on the matter, I thought I could add something to the discussion.

A few things about me: From 2000 to 2007 I worked as a programmer for Dgital Technology International (DTI), one of the leading newspaper software vendors (editorial and advertising) in North America and Europe. I worked primarily on editorial products. I have made roughly 25 visits to customer sites to perform troubleshooting or assist with software installations or upgrades. So I have an insiders look about how newspapers operate, albeit with a technical slant. I never learned much about how newspapers are run though. That kind of knowledge would have made writing this easier. If there are any inaccuracies (and there probably are), feel free to call me out on them, either in the comments or by way of email.

What is happening to newspapers?
To sum it up, the business is drying up. Circulation numbers are in decline. Whenever this happens, advertisers tend to pull back. The result is a net loss of revenue.

But why is this happening now? Can it be blamed on the recession? I was around during the last recession. There was a lot of talk in 2001 and 2002 about newspapers going extinct then, since advertising rates had fallen and circulation was in decline, much like it is now. In fact, the biggest difference between now and then is the severity of the problem. For me, watching the Rocky Mountain News shut down operations really opened my eyes to the magnitude of the problem being faced by print journalists.

So if the slump can be blamed on the economy that would mean that circulation would probably not have declined during the good times. Right? Let's check the numbers. I'll use the New York Times as a metric, even though it's not a good metric for the entire industry. Here are the ups and downs of circulation starting in 1998. I got these numbers from the NYTimes corporate site.

1999: +2.2%,
2000: +1.3%,
2001: +0.1%,
2002: +3.8%,
2003: -5.3 %,
2004: +0.3%,
2005: +0.2%,
2006: +0.5%,
2007: -1.9 %,
2008: -3.9%,
2009: -3.5%

What I see here is one good year, a few years of stagnation, and a several years (the most recent ones) of significant decline. I don't think the decline of readership can be based on economic factors; there must be something else.

Cultural shift?
Maybe we're changing as a people. Unscientific guessing tells me that we're working more and doing more things (after all, we're multi-taskers). We don't have time to read the paper in the morning or when we come home at night. Further, when we do make it home, there are so many other things available to occupy us. 100+ channels on the tube, a backlog of events on the TiVo and this miraculous internet invention all await us. There may be studies to support this, but I don't care. This is my hypothesis and I'm running with it: less people read newspapers because they have too many other things to do, and those things are far more interactive.

What can newspapers do?
There are two obvious solutions. The first is to cut costs. The second is to get more readers. Let's take a look at each one individually.

First, cutting costs... Running a newspaper isn't cheap. There are reporters, editors, more editors, press operators, delivery people, circulation people and advertising people. Some papers have gone web-only. That eliminates the press operators and delivery people, and with the right software-a good chunk of the circulation department too. Give a few good salespeople the right software and analytics and I think advertising and sales would be covered. Could they skimp on reporters and editors--the real content producers? I think so, to a degree. See, if newspapers focused on local news and stopped trying to compete with CNN.com and USA Today for national and world coverage, I think they would be able to own that niche for some time. Radio took a similar path on the advent of the television age when home listeners declined. They found a niche to occupy (the car) and did well there for years.

Since I am in the software business, I'll go ahead and say that the software could be better. I'm not referring to the quality of the software, but rather what it does and how it used. For instance, at DTI we had release cycles of around 6 months. Imagine waiting 6 months to see something like Twitter or Facebook integration, or integrating the latest Adobe CS suite. Shortening that cycle would have been difficult from quality and cost perspectives. By "better" I mean more nimble. Smaller pieces working together instead of a monolithic suite. It would be hard to do, but I think it could be done.

Next is readers: how do they get more readers? My first suggestion is to reach out to them wherever they are--not just on their front lawns. Smart phone adoption is on the rise. I get a good chunk of news through my T-Mobile G1. It is easy to scan my syndication feeds when I'm in line for something. But I don't pay for it, which means nothing is going back to the paper, or whoever generated the content in the first place. So increasing readership alone isn't going to keep the papers afloat.

I just don't see myself paying for news. The Internet has made me used to having things free and my way, and I think a lot of people are with me on this one.

But this content has value, right? Surely, someone is willing to pay for it. I mean, if the switch were turned off today and all news went away, there would be a hole in my life. Americans need and crave news coverage.

As in the past, I think the advertisers will step up. But they're probably going to want to know a little bit about you, to, you know, make sure you get the right ad. Consumers benefit from this. Even if you routinely ignore all adversing, under this scenario, you're ignoring advertising targeted to you for stuff that might actually fit your interests. I have found myself, on occasion, clicking on Google ads, even though it rarely results in me buying something. The process works.

My predictions
In ten years, there will still be newspapers. But they'll be owned by local television stations, who will have taken over as the main supplier of news content. This suits me: their stories are shorter and I think that fits the lifestyles of more people nowadays. These newspapers will be free and contain copy taken from the newscasts and modified slightly for print. The ads in the papers will be the result of TV spot upsells. There won't be classifieds since you can't execute a full-text search on a newspaper.

This model cannot currently exist in the United States because laws prevent interests that own one type of media outlet from controlling another type. But that will change soon, if it hasn't already started. (Tracking the status of this without becoming an expert is well nigh impossible.)

I don't know much about what kind of software TV stations use to handle their content (I know a bit about the advertising side), but I seriously doubt it would integrate well with print content at this point. The disciplines haven't converged enough yet.

It will be fun to follow this story for a few years to see how things pan out.

Simplifying the Trip

2009-06-14T21:07:00.000-06:00

Our vacation starts soon. We're going some place far away by way of jet.

We've had a Sony MiniDV camcorder for almost 10 years. It still functions perfectly, but with four children in tow, it has started to feel more like an anchor than anything else. The bug bit last summer and I bought a smallish camcorder that recorded directly to a hard drive. Two days later with a bit of buyers remorse and the utter realization that the camera was completely incompatible with the MacOS (even though the box claimed otherwise) I returned it.

I decided to be a bit more pragmatic about it this year. After some reasearch I realized that I didn't want to spend good money on a small SD camcorder knowing that I would probably want a HD recorder in the next 5 years. But I'm not ready for an HD recorder yet either (I don't have the storage capacity).

So what do I do?

It didn't take long for me to come across the Sony Webbie and the Flip Ultra HD. These are small devices that record directly to flash in HD mpeg-4 format, priced in the $150-$200 range. Now don't be fooled by the "HD" in the product names. While both devices do record HD video, they do so using a codec that his highly compressed and lossy (think youtube HD quality). They don't do well in low-light situations either.

All this got me to wondering what the video quality on my digital camera, a two year-old Sony DSC-T20, was like. Turns out, it records 640x480 mpeg-4 at 30fps. This falls in the range of "good enough for me" and it's technically HD like the Sony anf Flip. After all, I'm not going to use it to record piano recitals, awards ceremonies or new babies. I just need something good enough to capture the cute moments that I want to remember when I'm on a trip. And let's face it--Nicole and I only ever go back and watch these videos once or twice a year anyway. (Maybe they will become more important as we age.)

So it's settled--I'm going to use my digital camera to record video on this trip. I did decide to spring for a larger memory stick though--8GB where there were only 2GB before, and a second battery for hot swapping.

Side question: I wonder how long before these devices converge? Tangential jab: And will it include a cell phone and app store as well?

Stupid CSS Tricks

2009-06-03T22:33:00.000-06:00

I have recently made the committment to wrap my head around CSS. In other words, I'm tired of guessing. Part of this experience will have me documenting my discoveries on this blog.

Here is what I learned today:

Pseudo-classes aren't just for anchors.

That's right. You're supposed to be able to apply standard pseudo-classes to just about any selector you can come up with. This means you don't to rely on a) jQuery or b) onmouseover/onmouseout to do the hover-effect work for you, so long as you can do it all in CSS.

Here is some sample code:

<html>
    <head>
        <style type="text/css">
  
        /* a class for an element */
        span.my_hover_class:hover {
            background-color:red;
            cursor:pointer;
        }

        /* attached to nested elements. */
        div > div > span > span:hover {
            background-color:green;
            cursor:pointer;
        }
 
        </style>
    </head>
    <body>
        <span class="my_hover_class">Should hover red (span.my_hover_class)</span>
  
        <div>
            Should not have hover effect.
            <div>
                <span><span>Should hover green (div>div>span>span)<span><span>
            </div>
        </div>
    </body>

</html>

I would like to show the effect here, but the Blogger editor insists on tidying up my pasted HTML. I uploaded the file to my website.

This example was tested in Firefox and Safari on a Mac.

Tagfriendly Updates, 1 Jun 2009

2009-06-01T22:00:00.000-06:00

I've been quietly hacking away, albeit slowly, on the next iteration of my Tagfriendly project. The back end is in a state I am happy with, so this version is all about making the front end suck less.

You can sample the work as it progresses at http://www.tagfriendly.com/tf2. This is truly alpha-quality work, lacking much in the way of CSS, but I link here because I know a few people follow this blog and are interested in it. That, and since I'm not actively working on the current version, I feel the need to show something for my efforts. :) I push updated code live about once or twice a week.

I use more AJAX this time around, including fancy transitions and assembling the page in chunks. I'm not sure if this is the right approach yet or not; I'll see how far it gets me.

In a few days I'll start work on the new player, now that I've discovered there is a published API for the Yahoo! Media Player (that is very hard to find via google).

And Now, A Completely Unrelated Gripe:

I offcially hate eBay. I sell one or two items on eBay per year, so things are usually in a different place every time I need to sell something. I can handle that, but two things tonight really bugged me.

1. I could not find a way to sell my stuff without signing up for and agreeing to automatic payment of seller fees. This needless hoop was in my face and prevented me from doing what I needed to do.

2. I have an unopened seasons 1-5 boxed set of 7th Heaven that I want to get rid of.** Well, in its wisdom, eBay has decided to cap shipping on DVDs at $3. In other words, I couldn't charge more than $3 if I wanted to. But this boxed set weighs about 5 pounds and will easily cost me $6-7 ship. I realize what they're trying to do, which is crack down on S+H abuse, but at what cost? They know the exact item I'm selling because they had me enter the SKU. They should know how much it weights and what it would cost to ship through various methods. Total fail, eBay; it's craigslist from here on out.

**I really thought Nicole liked the show and it would make an excellent Christmas gift. It turns out she only watches the show on Myth because she can skip the commercials and watch it at 150% speed. Total fail, Gary.

2009-05-30T07:00:00.000-06:00

I don't blog about music as much as I would like. I used to do it a lot.

I listen to music all day, just about every day. I subscribe to *a lot* of music blogs. So much that I decided to create my own blog aggregator to help me keep track of things (Tagfriendly.com, for those of you who haven't been there yet. You can create music-related RSS feeds based on your own search criteria.).

So the problem is not that I have little to say about new music, or the music scene in general, but that I have *so much* to say. If I didn't have to make a living doing Other Things, I could write about music (not that anybody would care to read it) all day, every day. Watching the indie scene blossom over the last five or six years with the help of the Internet has been a special thing for me. I listen to more music now than I ever have. And hey, record executives: I buy more music than I ever did too (my wife will attest). Even so, I know that I am just moving slowly across one facet of a very large musical stone. And that is a good thing--I won't be getting bored any time soon.

To that end, I have decided to carve out a few minutes each day and devote them to writing about music. Some of it will end up on this here blog, hopefully as entries about new music, or music that is at least new to me.

Let's get started...

Some good music that you're probably not listening to

"Oh My God" by Ida Maria
I thought it was Clap Your Hands Say Yeah at first, but it wasn't. This is an honest song sung urgently almost desperately, and right on the verge of losing it. "oh you think I'm in control... oh you think it's all for fun". Moving and tender ("find a cure for my life... put a price on my soul"), but dishing it out at the same time. This is the kind of song that makes it hard to stay in my chair. (wikipedia link)

"In the Night" by Basia Bulat
That is the sound of an autoharp, if you're wondering. Organic and edgy, pop-folk for the aughts (what will we call this decade?). I've been saying for several years now that the best new music is coming out of Scotland and Canada. Bulat represents the latter, hailing from Ontario. I like this song because it makes me feel good. It carries a message about rising up above struggles ("Storm and shadows fall to pieces / to my heart like a comet / carry so that I can / soar like an eagle"). And while it's still possible for struggles to keep us down ("sometimes it takes the night to fall"), it doesn't have to be that way all the time. (wikipedia link)

"New Moon" by Sambassadeur
When it comes to twee, you can't beat Tweeden. Er... Sweden. Sambassadeur are utterly forgettable, but still very pleasant (think: "spring time"). Proof that good music doesn't have to captivate or mesmerize; it just has to not make me throw up in my mouth. I really don't know much about this band. I suppose they are few Swedish twenty-somethings who will be around for a few years before moving on to meatier projects. (wikipedia link)

Yes Virginia, there is a garden...

2009-05-21T18:00:00.001-06:00

So many domestic projects and assignments lately leaves little time for writing about other things. Some of those projects have me out in the yard though, so things could be worse.

During the first week in April I started making trips to the garden to assess things. To my surprise, I discovered that my artichokes from last year survived the winter. (Look at the picture!)

I first planted artichokes three seasons ago. Starting from seeds I ended up with two plants that year. Neither of them flowered though. I knew that it would take two seasons for the plants to mature, so I bedded them down well (I thought) and let them have the winter off.

They died.

So I started the process over last year. For some reason I ended up with a lot more plants. Seven of them. Every seed I planted (from the same seed pouch as the year before) germinated. The season ended, winter came and I bedded the plants down again. I used newspaper and tree leaves, same as I did the year before.

They made it this time. This means that I should have a really good crop of artichokes this year. Nicole and I are both looking forward to them.

I planted corn this year, for the first time ever. Normally I skip corn because it is so easy to buy and, to be honest, it would block out a lot of sun other plants in my garden could be using. To keep me interested, I opted for an heirloom popcorn with a shorter habit. The kids are really digging the fact that we are growing popcorn in our garden this year. I noticed the first shoots (corn is a monocot!) poking up on Monday.

Beans are in the ground, a week late, but should be sprouting any time now. Bush beans this year--I'm tired of managing the trellis.

Peas were planted at the end of April and are about a foot tall now.

I had to buy tomato starts for the first time in four or five years. I started my seeds in peat pellets as I normally do. Then I transformed them to styrofoam cups filled with a starter mix, again, as I normally do. I couldn't find my usual starter mix, so I had to use an off brand. That was a mistake. The transplants struggled after that, and really struggled when I began to harden them off a few weeks later. I've all but given up on them now.

We're doing several different kinds of winter squash this year. Giant pumpkins too. The aim this year is to keep it low maintenance, as we're taking a looooooong vacation in the middle of the summer. Hopefully it will take care of itself, so long as the watering system holds out.

Tweets and Scrobbling

2009-05-05T21:49:00.000-06:00

My hobby site, Tagfriendly, got with the times last weekend and started using Twitter. It tweets new songs as they are found, at a rate of once every 5 minutes.

Kind of noisy I thought, until people started following. TF followers are, for the most part, record labels. But there are others who are just interested, and others who are just... well, I'm not sure. I'm still trying to figure out how the underbelly of Twitter operates.

I also took time to get some deeper Last.fm integration put together. If you allow it, Tagfriendly can scrobble the songs you love/ban while you are listening to them inside of the TF player. My next step is to give the TF player some UI love so that it can pop out to be its own window and do a few other fancy things.

I took a quick peek to see what it would take to scrobble all songs played, but it would require the user to give TF his password. I'm not comfortable with that, so until there is another way, TF won't be scrobbling played tracks.

I'm going to focus a bit more on the player, as I think it can make the site more fun.

Hacking the Yahoo! Media Player

2009-04-14T07:00:00.002-06:00

I've been using the Yahoo! Media Player to stream Mp3s at Tagfriendly. Minor glitches aside*, the only real complaints I have is that it isn't skinnable and there is no public API.

About a month ago I started prodding it to see what would make it squeak. This post documents some of what I've found.

Disclaimer: I skimmed over the YMP terms of service and don't believe I'm breaking any of the rules. You should also know that YMP is a) beta software, and b) a hosted application. This means that any code you write or use that relies on specific methods or objects will be brittle and prone to breaking when Yahoo! releases updates. That said, you're on your own; I didn't make you do anything.

YMP is one of those nifty internet tools you can use by simply embedding a <script> element in your markup. If the page you've embedded it in contains links to mp3s, or if it links to an XSPF playlist, YMP picks it up and makes those MP3s streamable. This is powerful if you're a non-technical blogger and want to have an embedded player on your site.

My aim is to leverage all of that, but to take the Yahoo! face off and give it my own. I use jQuery to manipulate the Tagfriendly DOM, but really, any decent JS toolkit should allow you to get the same results.

The first thing you need to do is know when YMP is finished loading so that you can tell the UI to go away. Due to the fact that the Javascript you embed includes a bootstrap that downloads other things, you can't count on YMP to be ready when your document is. This is easily accomplished with a JS timer that polls to check whether or not YAHOO.MediaPlayer.setPlayerViewState is defined. When it is, call YAHOO.MediaPlayer.setPlayerViewState(YAHOO.mediaplayer.View.DisplayState.HIDDEN); to make the YMP user interface go away.

Now you are in the drivers set to start working with the MediaPlayer object. Here are some useful methods:

YAHOO.MediaPlayer.getTrackPosition()
     Gets the position offset (in seconds) of the currently playing (or paused) track.

YAHOO.MediaPlayer.getTrackDuration()
   Gets the track duration (in seconds). YMP appears to grab this from ID3 tags in the mp3, so you can't always count on this piece of data to be there.

YAHOO.MediaPlayer.play()
   Tells the player to play the currently queued song.

YAHOO.MediaPlayer.pause()
     Tells the player to pause the currently playing song.

YAHOO.MediaPlayer.previous()
     Go back to the previous song.

YAHOO.MediaPlayer.controller.EventManager.onNextRequest.fire()
     I found it odd that there was no YAHOO.MediaPlayer.next() method. This method does what you expect the non-existent next() would. There are corresponding fire() objects for play, pause and previous as well.

YAHOO.MediaPlayer.getMetaData()
   Returns an object that describes the currently queued song. Useful properties there, gathered from ID3 and the XSPF, include 'title' and 'artistName'. There're more if you care to look.

That's basically it--all you need to subvert the YMP UI and handle things your own way. This really is a guerrila API hack, as I don't think the YMP designers intended a public API. You can see the results on the front page of Tagfriendly. I went as far as providing a progress bar that gets updated as the song plays. The Javascript source is freely available too.

In the future I plan on displaying cover art and linking the songs to their Tagfriendly description pages as well as to music stores.

* I suppose not so minor because it really bothers me: YMP breaks down when you try to use a locally hosted XSPF file or the machine it is hosted on is stuck behind NAT. Firebug reports that YMP executes a GET with the URI to the XSPF. The backend of that GET does some data-munging of the XSPF contents. The results contain a basic JSONified playlist. The only problem is that if the [development] server that hosts the XSPF is behind NAT, that backend can't fetch anything.

Book Review: The Mystery of the Aleph

2009-03-27T05:58:00.002-06:00

I am a sucker for a good math book. No, not the calculus book you had to buy in college--I'm not referring to that. I suppose I might more properly say that I'm a sucker for a good book about math history. My first exposure to this genre came in high school when I read Asimov on Numbers. I was not aware then that Asimov wrote non-fiction, but at that point (I was 17, a senior) I barely had time to be aware of the mandatory reading assignments I was given (thank you, Milton, Ibsen and Faulkner).

So it started, and I've read a great number of math books since. A few weeks ago I picked up The Mystery of the Aleph: Mathematics, the Kabbalah, and the Search for Infinity by Amir D. Aczel. It is about the mathematicians who have studied infinity and the continuum. Infinity is one of those mathematical concepts that, as a beginner, one thinks he has his head around until he is taught a few advanced concepts. And then, of course, one realizes he knows almost nothing and it is difficult to trust what you already think you know.

The book focuses mostly on work done by Georg Cantor and Kurt Gödel, and explains how each lost his sanity during the course of trying to prove different properties related to the continuum hypothesis. While never able to prove or disprove the continuum hypothesis, each man ended up making valuable contributions to mathematics. Other disciplines too: as a computer science major I remember learning about how Gödel was able to prove that some problems are not solvable in a given system.

For the most part, I enjoyed this book. The math, when presented is covered in terms most non-math types can understand. There are helpful diagrams that serve to illustrate and clarify difficult points and concepts. I've read other works by the author and this book is about what I expected.

Something I didn't care for was the tendency to bring religion into the story where it didn't really fit. I couldn't see the point of dwelling on Cantors Jewish heritage. It didn't help the story and almost made me wonder if the author was trying to say "look, Jews have made important contributions to mathematics!" Big deal. The only group of people who haven't made important contributions to mathematics are the eskimos, and they're too busy to care.

So, if you want to learn about how our perception of infinity has changed over the years, go read this book. It's not bad.

Emotional Investing

2009-03-19T20:25:00.004-06:00

Nicole and I take X percent of our income each year and put in various retirement investments. We had a bit left over for 2008 and I've been wanting to invest in wind energy. Specifically, I'm looking for a wind turbine manufacturer who is well-established elsewhere and trying to break into the burgeoning U.S. market. (If you find such a company, and it can be easily traded in the U.S., let me know.)

While researching this I came across PWND. The techie in my wanted to snatch it up just so I could say I have stock in PWND. I mean, that would be l33t, wouldn't it? Anyway, I had to pass it up. What little I know about investing in stocks has taught me to stay away from ETFs.

Back to the research...

Restoring Myth Programs Whilst Maintaining Sanity

2009-02-22T09:46:00.011-07:00

My MythTv system has been running continually, more or less, since October 2006. Not too long, but long enough. Long enough, in fact, for cruft to creep in. I discovered this on Saturday when I went to restore data to the replacement for the myth drive affectionately known as "xfs2" that died about a month ago.

That drive had a capacity of 250GB. My myth data is important to me, but not that important. I had it set to back up once a month to another computer in the house via rsync. I didn't pay too close attention to the way I used rsync though--instead of removing the remote files that had been removed locally, they just stayed there. In other words, unneeded data on the backup never went away, even when I deleted shows on the myth system.

So when I went to copy 274GB of data to a drive that would only hold 238GB, things went haywire.

Thankfully, I am a programmer. I am equipped for these kinds of situations.

I needed to figure out which shows missing locally that were present on the backups. These are the files that needed to be restored. After that, I figured it would be gravy if I could remove the orphans that were in the database but not on any file system, be it local or backup.

It turns out that python is incredibly easy to use with MySQL. I created a simple program that would restore from my backup and also give me a list of orphaned programs.

Seems easy, right? The complexity comes when both local and backup storage are strewn across different drives, directories and hosts. I made it simpler by using smbmount to mount the backup system so it appeared more or less local. After that it became a matter of letting the script run* and then cleaning up the orphans with a simple sql statement.

Source code for myth.py is at the bottom of this post.

* This turned out to be tricky. Something on my myth host causes it to intermittently hang when copying files to or from a remote host. It could be my router for all I know. I do know that *any* keystroke received by the myth system causes it to wake up and start accepting traffic again. Meanwhile, the system clock thinks nothing has happened and starts up again at the same tick where it fell asleep--so the clock is off. This problem first started when I decided to upgrade to Ubuntu 8.10 and MythTv 0.21 on the same day. Truly disturbing, I know. I circumvent this by bandwidth-limiting scp to 2Mbit for the transfer. Transfers take longer, but at least they complete. And yes, I am retiring this system in a few weeks. :)


import MySQLdb as db
import os
import sys

existing_video_dirs = ["/xfs1/myth/video", "/xfs2/myth/video"]
# If you use scp, backup_video_dirs and backup_scp_paths need to be maintained
# in parallel. For sure though, backup_video_dirs need to be locally mounted.
backup_video_dirs = ["/mnt/remote_backups/myth_backup/xfs1/myth/video", "/mnt/remote_backups/myth_backup/xfs2/myth/video"]
backup_scp_paths = ["garyd@child:/mnt/xfs3/myth_backup/xfs1/myth/video", "garyd@child:/mnt/xfs3/myth_backup/xfs2/myth/video"]
backup_info_dict = zip(backup_video_dirs, backup_scp_paths)
possible_file_extensions = ["mpg", "nuv"]

# this is the place video gets copied to.
restore_path = "/xfs2/myth/video/"

# connect to the database and get the list of shows.
con = db.connect(host="localhost", user="root", passwd="root", db="mythconverg")
cur = con.cursor()
cur.execute("select chanid, starttime, title from recordedprogram order by chanid, starttime")
rows = cur.fetchall()

# keep track of the number that are there or not.
there = 0
not_there = 0
restored = 0

for row in rows:
    # name of the file is based on row values, plus a file extension.
    fname = "%d_%s" % (row[0], row[1].strftime("%Y%m%d%H%M%S"))
    # see if the file exists in the local myth dirs. If it does, there is no
    # action.
    exists = False
    for dir in existing_video_dirs:
        for ext in possible_file_extensions:
            path = os.path.join(dir, fname + "." + ext)
            if os.path.exists(path):
                exists = True
    if exists:
        there += 1
    else:
        # if the file does not exist locally, look at the backup directories to
        # see if it is there.
        not_there += 1
        backup_exists = False
        backup_at = None # path of backup file.
        backup_scp = None # scp path of backup file.
        #for tup in backup_info_dict:
        for dir, scp in backup_info_dict:
            for ext in possible_file_extensions:
                path = os.path.join(dir, fname + "." + ext)
                if os.path.exists(path):
                    # found a backup! use wild cards so that the video file
                    # and its preview get copied.
                    backup_at = path + "*"
                    backup_scp = scp + "/" + fname + "." + ext + "*"
                    backup_exists = True
        if not backup_exists:
            print "No backup for %s" % (fname)
        else:
            # move the backup to the live system. use cp or scp depending on
            # your preference.
            cmd = "cp %s %s" % (backup_at, restore_path)
            # something on my system is jacked. I need to bandwidth limit scp
            # or else the myth server stalls.
            cmd = "scp -l16000 %s %s" % (backup_scp, restore_path)
            print cmd
            os.popen(cmd)
            restored += 1

# output the stats.
print "there:%d not_there:%d restored:%d" % (there, not_there, restored)

Update: fixed tabbing in code. Should have used pastebin.

Tagfriendly Update

2009-02-21T07:17:00.001-07:00

I am going to tie off version 0.1 since I've accomplished the meager goals I set for myself in January with regard to Tagfriendly. Here is what they were:

Track music blogs.
Full-text search.
Dynamic RSS feeds (turned on last night).
Learn python (still a work-in-progress).

I've added other features along the way, but those are the basics that I wanted. Here are some things I've been thinking about for the next iteration (roughly one month):

More Last.fm (social) integration. Lots of ideas here. For example:

What if when you scanned the recent MP3s, you could know which ones your friends like?
What if you could create an RSS feed that would generate entries related to your top 10 (20 or 30) songs or bands? (this was suggested by a friend.)
What if you could scrobble love/hate directly from the website?
Lots of cool things here, and they won't be too hard to implement.
The problem with Last.fm is that using them has a detrimental impact on Tagfriendly site performance (their services are S-L-O-W). I've used a lot of caching (artist information mostly), but that only gets me so far.

Simple identity management. This will enable features like "show me all the new stuff since the last time I was here." I'll start with just a cookie. That has limitations, so I'll eventually push for simple email registration.
Flash based in-browser mp3 player to stream the songs from the blogs. The ethics of this are probably dubious, but I have noticed that many of the bloggers deep link to mp3s stored on other sites. I guess we're all in this together. Hopefully I can find something off the shelf that does this.
I need to clean up the back-end scripts that do the crawling and generate the static pages.
Better tests.
More blogs. I'm up to 30 and crawling+indexing get done in about 3-4 minutes. I'd like to scale up to 100 blogs for this iteration and see how things are handled.

That's about it for a month. I average about an hour a day of Tagfriendly development, so this is about the right amount of work for a month. (Honestly, I could probably spend an entire month on testing, but how sexy is that?)

The Saddest Story Ever Told

2009-02-18T18:00:00.000-07:00

If you depend on a computer like I do, this one will make your innards shrivel up inside you and convince you to become a hermit...

The display unit in my company-owned MacBookPro when belly up on Monday night (thank you, Nvidia), one day before the one-year warranty would expire. (This was actually fortuitous, as my employer did spring for Applecare, but it was lost and never registered to my Macbook. Another story.) I quickly made plans to take the ailing machine to the Apple Store in Salt Lake City on Tuesday. Then I set about deciding on how I would work until I got it back. I am a remote employee; no IT staff to give me a loaner when equipment goes bad.

Here is the lay of the computers in my house:
1. Nicole has a 2.5 year old white Macbook (single core Intel 1.2 GHz?)
2. The kids have a brand new dual core Intel Atom 1.6GHz machine. It has a 1.5TB drive in it and serves mainly as a network storage device.
3. MythTv runs on a very old AMD Athlon 1.3 GHz machine.

Using the Myth server was out of the question. Not only is it too old to be useful in my work, it has reached mission-critical status in our household. We need our TV shows.

Nicole offered, and I breifly considered using her computer. I decided not to because I thought the kids computer might be a bit more powerful (though not a mac) and I didn't want to bother setting up a my dev environment there. You see, I had a plan.

The Plan involved using the kids computer. It was new, though underpowered, but I figured I could make it work fine. Before taking my macbook in I made a copy of the windows virtual machine I sometimes work in. I figured I could install VMWare Player on the kids computer, turn on the VM and I'd be in good shape. Setting up a non-development machine to do development work usually consumes about half a day for me, so I was keen on avoiding this.

But it wasn't meant to be.

The only file system on their computer that had space for the VM was formatted XFS, and I had been having little problems with it that I had mostly chosen to ignore. VMWare started complaining about 30 seconds after bringing the VM up. Then the OS dropped the volume completely. Ouch.

I ran xfs_repair and re-mounted the volume to try again. I'm a hopeful person and besides that, I needed to get some work done. Same exact problem. Ugh.

At this point my thoughts were focused on two things: First, I had a few work tasks that must get accomplished, and I still didn't have a dev environment to do those tasks. Second, I needed to get rid of XFS on that volume. It would not be viable in the short- or long-term.

First things first... I still haven't put the new 250GB hard drive in the myth server to replace the dead one. I decided to stick a USB enclosure on it, format it as ext3 and move the windows VM to it. At least then I could get into the VM without it crashing. The VM ended up performing quite poorly (due to running over USB), but would be ok for an afternoon of work.

Next I had to figure out how to get rid of the crappy XFS partition that was giving me grief. The only problem was that it had about 750GB worth of data that I couldn't just delete. It contained my music backups (expendable), DVD rips (not very expendable, but I could live without them), myth backups (needed for a full recovery if I ever got around to replacing the dead drive in the myth machine) and other assorted backups.

Luckly, I happened to have an extra 1.5TB drive laying around. I plan to put together a new Myth server in a few weeks and have started assembling the new hardware. I briefly tried installing windows XP on that drive (wasn't looking forward to working in a VM), but the aged XP installer didn't like the humongous drove. Oh well. You know the drill by now: USB enclosure, format ext3, and move data off the XFS volume. Ubuntu estimated it would take about 8 hours for the transfer, so I went to bed.

In the morning, the copy looked good, so I removed the USB drive and set it aside. Next I unmounted the XFS partition and reformatted it as ext3. Then I moved my 30 GB VM onto it and fired it up.

It works a lot better. Not only is it stable, but it runs a lot faster (this was expected). I suppose I can work like this for a week until my Macbook comes back.

The next step will be to restore the backup data off the USB drive back onto the backup machine. I'm losing the stomach for hardware maintenance, so I'm going to put it off for a day or two.

NOTE: I've used every model of 15-inch MacBookPro/PowerBook since the Aluminum G4. Each one has needed repairs within the first year (one even shipped with a defective trackpad). My take: always spring for the extended warranty.

Full-text Indexing on the Cheap

2009-02-06T07:00:00.005-07:00

After Christmas I decided I wanted to build an mp3 blog aggregator that would be all my own. I've decided to chronicle the process here on my blog.

I have been a big fan of Lucene for years, although I now prefer Solr (it's still Lucene) for new projects. The main reason I've stuck to Lucene is that most projects I've worked on have been Java-related and limited hardware resources haven't been an issue.

All that changed with the latest incarnation of Tagfriendly. I develop it on a linux VM, which has all the hardware resources of my workstation (more than enough to handle Lucene), but I deploy Tagfriendly to a VPS where my VM is one of many that may be sharing the same physical hardware. To put it simply: I am hardware-constrained, currently stick with 128MB RAM and 5GB of disk space. For example, consider this: apt-get update craps out if I'm running lighttpd+paster. Yeah, it's like that.

While I can grow to a bigger VPS if I need to, I figured it would be good to build Tagfriendly to be lean on resources. Bottom line: I couldn't consider using Lucene as the search engine. So I started to look at other options.

The first two I considered were tsearch2 and sphinx.

tsearch2 kind of surpised me. The last time I looked into it was several years ago when it was a postgres contrib module--not part of the standard release. Even though the idea of having a full-text search structured like a SQL query was alluring, I didn't consider tsearch2 for long. I knew I wanted to be able throw just about anything into the index and rely on a "type" field to restrict results to particular data (e.g., mp3s or blog entries).

That left me with sphinx. I came across it while I was googling for Lucene alternatives. It has python bindings, is written in C, and claimed to be very lightweight. Seemed like a good fit. One downside I noticed is that once a sphinx index is created and initially filled, it can't be added to. That means I would have to generated the index from scratch every time I need to update it. Ouch. (Maybe this isn't true in newer releases.) I kept Sphinx on the list, but kept looking.

A day or two later, before I had written any indexing code, a friend pointed me to Xapian, which has a lot of the characteristics of Sphinx, except that Xapian is more mature (and can be updated). I settled on using Xappy which is a python wrapper around the basic python bindings provided by Xapian. Xappy simplifies the fielded aspect of managing a Xapian index (makes it feel more like Lucene). After playing with both the standard python Xapian bindings and xappy, I recommend using Xappy for simple indexing projects. The learning curve for the traditional Xapian API is pretty steep.

A few hours after finding Xappy (spread out over a few nights of work), I managed to produce a simple index page in pylons and an indexing daemon for Tagfriendly. The index only contains about 1500 documents at this point, so searching is still quite snappy. We'll see how it goes as the document count grows.

With an index in place, I am free to move on to some of the really interesting features and ideas I want to explore with Tagfriendly. For example, I am excited to implement a feature where users can create RSS feeds based on custom search criteria (e.g. includes Morrissey, but not The Smiths).

Superbowl Notes, 2009

2009-02-02T06:12:00.004-07:00

I had taken pains to make sure watching the super bowl would be simple and painless. This included having dinner ready by 4:30 (we get home from church at 4), cookie dough (for snacks) already made, and Mythtv set to record the whole thing starting at 4.

We were done with dinner at 5. I took my time cleaning up, even though Gavan was chomping at the bit, unable to wait for us to start watching the game. I started a batch of cookies and went downstairs to start watching with Gavan.

I'll interject here and say that I'm not a huge TV sports fan. I was in my early years, but those tendencies seem to have left me about the same time fatherhood arrived. (I'm not sure if there is a connection or not.) The truth is, I would have peeked in on the game at intervals, watching for perhaps five minutes at a time. In the mean time, Gavan would be riveted to the sofa, basking in every moment of it. Such is youth.

None of it was to be though. Our one-year old, while Nicole was retrieving some items from the crawlspace, where the myth server is kept, managed to press the exact sequence of keys required to send the machine into hibernate. While it was recording. The super bowl.

If you follow this blog, you know that we've had a few problems with our MythTv setup lately. Apparently, this was too much for the system. Bringing the machine out of hibernate while in the middle of a write (recording the show), and a read (we were watching it on the MVP) was a Very Bad Thing, at least to the poor XFS filesystem that received the battering. I had no choice but to bring the system down. In the process of bringing the system back up again, I was informed by fsck that my disk was in bad shape and that I wouldn't be using it until I personally rectified the situation. Ouch. That is two of two disks down for the count.

All I really wanted to do was watch a few minutes of football with my son.

I grabbed the nearest piece of long coax, intending to connect it from the dish receiver--in the crawlspace, to the television--in the basement. I was angry. And not just a little bit angry. My daughter's Hello Kitty kick-ball was the nearest object I thought I could kick without injuring anybody, so I took a swing at it. And when I say "took a swing at it," I really mean to say is that I kicked at it like it was the last kick an unreformed kicking addict would ever kick because it was the last kickable object left in the universe. I put my heart, soul and body into it.

I mostly missed the ball. And mostly kicked the door. Ouch. I'm not doing good here. My foot throbbing and turing grurple (green+purple) in spots, I connected the coax, made sure the game was on, and then let Gavan know to come downstairs.

It was halftime at this point. I retreated to a bathroom to let some cold water run over my foot. I helped some. I was sure I had broke something.

The pain subsiding, and firmly in control of myself again, I sat down at the console to take a look at the Myth system. I set up a temporary space to hold shows and brought it back up, sans 500BG of storage. Ugh. But at least it was recording. I switched the coax back and all was right and sane again.

I'm kind of glad I missed the first half because the second half was 30 minutes of the best football I had ever watched in my life. Coming back from 13 points behind is difficult enough. In fact, the only thing more difficult, I'd say, is coming back after letting a 13 point lead slip away. And I saw both of those things last night. Truly remarkable. In my heart, I was silently rooting for Arizona, the underdogs. I was elated as I watched Larry Fitzgerald streak 80 yards down the field to put Arizona up. And then a few minutes later, in awe as I saw a Steelers team, firmly in control, take the ball 99 yards down the field and score on Santonio Holmes pass. One of the best super bowls ever.

iTMS link generator API

2009-01-31T18:00:00.003-07:00

I set about looking for an API that utilizes the iTunes Music Store link maker, thinking they would be all over the place (deep links that is, and includes referral program ids). Five minutes of half-hearted googling didn't get me anywhere I needed to be, so I decided to take matters into my own hands. (Note: either I am a crappy googler, or the Apple legal team keeps these things off the air. Seriously, this code must have already been written 10 or 12 times now.)

A bit of poking around revealed that Apple (edgesuite, linkshare, linksynergy, whoever!) uses GET for their search forms. This makes writing a scraper API as complicated as parsing HTML, which is pretty easy in this day and age. I decided to use python, because I'm learning it by forcing myself to use it for all my extra-curricular projects.

I've released the source code under the code section at my website. The license is, uh... liberal. So don't be afraid to use it if you find it useful. It depends on the excellent BeautifulSoup library to do the heavy HTML lifting.

Here is some iTunes deep-linky goodness for you. These are the top 5 most played songs in my library. Buy them:
1. New Slang, by The Shins
2. Valley Winter Song, by Fountains of Wayne
3. Dreams Anymore, by The Magnetic Fields
4. Trust, by Gravenhurst
5. Saint Simon, by The Shins

BTW, I've resurrected Tagfriendly as the mp3 blog aggregator I've been working on, soon to have iTMS referral links. It isn't much, but I'm adding features all the time.

(Tagfriendly was, at one time, an automatic ID3 editing tool I created that mostly worked.)

Return of the Clone

2009-01-30T18:00:00.003-07:00

Any time you can mash-up references from Star Wars into a blog entry title that actually applies to the content, it is a Good Thing.

Over the Thanksgiving holiday I did a lot of infrastructure work on the computers that run in the house. Part of that work involved setting up a computer to be used by the kids that would also double as a file store for backups (1.5TB disk), upgrading the MythTv server to 0.21 and doubling its storage capacity to 500GB.

While I was at it, and since I had the space, I decided to set up a weekly rsync to clone ("mirror" is probably more appropriate) the myth video drives to the backup machine. I'm thankful for that now, as one of the myth drives died last night (250GB Western Digital). The drive was 3 years old and had spun more or less continually over that time. I figured its time had come.

I've ordered a replacement drive which should arrive early next week. Restoring things should be a simple (gosh, I hope!) as putting in the drive, formatting it, making sure fstab is happy, and then a reverse rsync. In the meantime, Myth is clipping along using what space is left on the remaining drive. It had to expire A LOT of stuff, but is doing fine now, except that it still tries to expire shows on the missing drive. (I wanted to make "phantom" reference here.)

I did go through the pains of removing the path of the broken drive from its storage group, as myth has a tendency to start recording shows in odd places if it can't find an intended storage group path. I had to drop down into sql to do this--I couldn't find a way using the UI that didn't involve deleting the entire storage group.

Mp3 Blog Aggregator: status and some code

2009-01-23T22:51:00.006-07:00

My last post was a lamentation about how the current set of mp3 blog aggregators don't do it for me, and at the same time a declaration that I would do something about it.

I've spent my spare moments this week hacking at the problem and it's starting to bear fruit.

The first of it is a simple id3 reader implemented in python. It simply reaches out over the tubes and grabs the id3 information from an mp3 that is hosted on a server somewhere. Nothing too complicated, except that it can be configured to extract any images that might be embedded in the mp3.

Knowledgeable readers might be asking: "why didn't he use one of the three or four existing python id3 libraries?" The answer is this: I planned on creating a blog crawler (mentioned later) and a website for this idea, and would do it all in python. As a warm-up exercise, I figured it would be good to create a simple id3 reader. I had already done it in Java, so it mainly became an act of seeing how the Java idioms I am currently used to translate over into python. (Note: if you bother to download and read the code, please be gentle. It's the first real python I've written. Feedback is appreciated too.)

The crawler is mostly done. It came together more quickly than I thought, although it still has rough edges. It runs a few times a day, notes new blog posts and gathers what information it finds into a database (postgres).

The website is where the work needs to be done now. I have gotten no further than creating a few simple query+display pages that I've been using to view results from the crawler. I've experimented with different ways to present data (entry-centric vs mp3-centric) and still haven't come up with something I like. I've got time though. And the longer I wait, the more useful data I'll have from the crawler.

I'm still using pylons for the website, although I had second thoughts after spending too much time fighting mako and the way it manhandled my nice unicode mp3 tags.

I have yet to tackle the problem of dynamic RSS generation, but I have some good ideas in my head for that.