Thoughts on Rdio

Clearly, each of the services get some things right.  I could go into detail about why I like Rdio and chose it over its competitors (Last.fm, Pandora and Slacker), but this post is mainly about why Rdio is right for me and a what would make it better.

The Good
Playlists
I like Rdio mainly because it gives me flexible listening options:

• It lets me have playlists with cherry-picked songs.
• It can generate playlists based on an artist.
• I can queue up entire albums.
• I can listen to songs offline using my android phone.

API
Although they didn't have it when I started my subscription, Rdio have added a public API, which is cool.  This means that developers will create interesting applications that use Rdio (unlike say, Pandora). I've played around with it a bit and it is pretty natural.

Discovery
I absolutely *love* that I can listen to entire albums at once.  Remember that band you heard that one song from once and you meant to go back and check them out?  Only, you had a terrible time finding a decent way to sample all their music and you weren't ready to make the commitment of actually paying for it.  (We have all made that mistake before.)  Rdio solves that problem: queue up their catalog and listen to everything for a day or two.  Then go buy, or not.

Room For Improvement
Offline Content

• I wish the desktop application had offline mode like the mobile app.  
• Managing offline media from the mobile application is cumbersome, especially if you have synced a lot of songs.  
• Managing media from the desktop application is even more difficult.
• It would be awesome if I could create a station based on an artist and then sync those songs (in much the same way I can create a playlist and sync those songs with one action).

Most of my grief with offline songs would be eliminated if there were a way to expire the offline content so I didn't have to manage it myself (like a DVR).  E.g.: keep this song for a [week, month, etc.], delete it after I play it [once, twice, three times, etc.]

Playlists
Rdio does not have "genius mode."  I would like to generate an awesome playlist based on a single song of my choosing (like iTunes).  They have artist-based playlists, but it isn't the same.  Theoretically, an enterprising programmer should be able to create software that exports an iTunes genius playlist (or any iTunes playlist) to an Rdio playlist.  Sounds like a fun weekend project.

I wish I could create multiple queues (e.g.: a "work" queue, a "home" queue, and a "quiet sunday" queue).  I could use playlists for this but I cannot add entire albums to playlist unless I add the songs one at a time.

The Rdio machine learning algorithms could take a lesson from Pandora or Last.fm.  They are not that good.

If you use queues and then go listen to another song, your currently playing queue item (an album) is lost.  This used to bite me a lot and annoyed me until I eventually changed my habits.  This could be resolved if Rdio just remember where I was in the queue when I play something adhoc.

Streaming
It takes too long to go from pushing the play button on my mobile player to actually hearing something.  If this is a problem of pushing bits, maybe Rdio could push some low-quality data (doesn't require as much bandwidth) at first to get some data on the users player quickly, then come in later with the full-quality audio.  If this is a latency problem, my bad.

Rating, Scrobbling and Data
It would be cool to scrobble love/hate from Rdio.  Currently I have to go to the Last.fm website to do it.  Also, it would be nice to rate songs (like iTunes) or tag them as I listen to them.  This is helpful for music discovery at times when I do not really want to be distracted by the process of copying down a song or artist name.

Something to Show For it All
This isn't a deal-breaker (none of it is, after all--I am paying for the service).  It saddens me that if I give Rdio $10 a month for the next 10 years and then stop, I will have nothing to show for it but the memories.  It would be great if I could keep some of the music permanently.

I realize that it is the same value proposition as cable TV and that I would have my cake and eat it too.  But folks--this is music.  I am used to keeping what I pay for.

Summary
I like Rdio.  Out of all the players so far, it works best for me.  It as the most flexible listening options of any streaming music service, but I think there is room for improvement.  I hope they keep up the good work.

RESTful Cassandra

A lot of people, when first learning about Cassandra, wonder why there isn't any easier (say, RESTful) way to perform operations.  I did.  It didn't take someone very long to point out that it mainly has to do with performance.  Cassandra spends a significant amount of resources marshaling data and Thrift currently does that very efficiently.

So I put away my RESTlessness.

I've heard more people lately clamoring for the feature, so I gave some thought about how I'd go about it.  One approach would be to wrap Thrift.  That would be nice from a coupling standpoint, but I think performance would be pretty crappy.  After all, it is just adding another layer of marshaling that needs to be done; nobody needs that.

I eventually arrived at the decision that an HTTP Cassandra Daemon/Server pair similar to the existing Avro and Thrift versions would do the trick.  It would basically be a straight port, with a few minor caveats.  One big thing is that HTTP uses a new connection for each request, so storing Cassandra session information in threadlocals is gone out the window.  This means that authentication needs to be abandoned, done with every request, or we need to use HTTP sessions.  Punt.

Today, while half-listening to some lectures at ICOODB I decided to see how hard it would be to throw something together.  I ended up with two classes containing stripped down implementations of get and set.  I pushed the whole thing to github if anybody is interested.  I used the built-in Sun HTTP service because I didn't want any extra dependencies and building services on top of it is pretty straightforward.  Results are returned in JSON format that should match what you would see if you used sstable2json to export data.

This is clearly a proof of concept, but I think it demonstrates that the idea is sound and could be implemented fairly quickly.  Maintenance would be another story.  One problem with maintaining the Cassandra Avro bindings is that they regularly get out of sync with what is capable using Thrift.  An HTTP Cassandra wrapper would suffer the same fate without an active champion.  I'm interested, but I'm not *that* interested.

Anyway, have fun.

-- DETAILS --
The following URI formats are expected:
/get/keyspace/column_family/row_id/super_column/column_start/column_end/consistency_level.  If you don't want to pass it in, leave it blank.  Empty strings are interpreted as null when appropriate.

Here is an example from my tests:  http://127.0.0.1:9160/get/Keyspace1/Standard1/1//10/11/ONE

The main thing to deduce here is that the super column is empty (see the double slash?).  If you haven't realized by now, I've gone ahead with the assumption that your keys and column names are strings.  This isn't good enough.  All the details we need to become type-aware are available in the comparator for the column family.  As a shortcut for now, you can append "?asString" to the end of the URI to have all byte[] values converted to strings.  Without it they are displayed as hex.

Updating works the same way: /set/keyspace/coumn_family/row_key/super_column/column/value/consistency_level

e.g.: http://127.0.0.1:9160/set/Keyspace1/Standard1/1//11/aaa/ONE


UPDATE: I went ahead and created two additional implementations that use jetty (bare and with servlets).  This generated a bit more code, but opens up the way to getting sophisticated with sessions.

Building Cassandra Thrift Bindings on OS X

A few weeks ago, I came to Rackspace to work full-time on Cassandra in their cloud division.  So far, I'm having fun and learning new things.  Apart from Cassandra, one of the projects I get to figure out is Thrift.  Thrift is a tool that allows you to define a service interface and then generate stubbed service bindings in different programming languages.  A programmer then takes the generated code and makes it do the things it is supposed to.  In an ideal world, this simplifies the process of, say, stubbing in a PHP client that can speak to a server stubbed in Java.

Right now, I'm the lone wolf in the office doing development on OS X.  The glitches so far have been minor, but I was forewarned that I might want to reconsider [using linux] when it came time to work with Thrift.  Well, that time started today.  I've been a faithful linux user for about 10 years, but I've been a faithful Mac user even longer.  I'm not ready to make the switch to full-time linux yet; I like my Mac.

Fortunately, Google was my friend when it came to figuring out the secrets of building Thrift on OS X.  Credit goes to Nathan Ostgard and his blog post for getting me going in the right direction.

1.  You definitely want to install macports.
2.  Install boost and log4j
sudo port install boost
sudo port install jakarta-log4j

3.  Download and install thrift
curl -o thrift.tgz "http://gitweb.thrift-rpc.org/?p=thrift.git;a=snapshot;h=HEAD;sf=tgz"
tar -xvf thrift.tgz
cd thrift
echo "thrift.extra.cpath = /opt/local/share/java/jakarta-log4j.jar" > ~/.thrift-build.properties
./bootstrap.sh
./configure --prefix=/opt/local

4.  You're going to get an error during configure:
./configure: line 16440: syntax error near unexpected token `MONO,'
./configure: line 16440: `  PKG_CHECK_MODULES(MONO, mono >= 2.0.0, net_3_5=yes, net_3_5=no)'

I couldn't figure out how to tell configure "no csharp, please" through the command line, so I just commented out lines 16439-16442 and ran configure again:

./configure --prefix=/opt/local

5.  You know the drill:
make
sudo make install

That's it for Thrift.  The next step is to generate the Cassandra client.  The Cassandra wiki has steps to generate a python client.  This works fine except that the thrift python module was installed to a place where the OS X python can't see it. You'll get the following error if you try to run Cassandra-remote:

Traceback (most recent call last):
  File "./Cassandra-remote", line 11, in
    from thrift.transport import TTransport
ImportError: No module named thrift.transport

There is probably a right way to fix this problem, a way that is right for OS X, but I had no patience.  I added the the following line to my ~/.profile:

export PYTHONPATH=/usr/lib/python2.6/site-packages

Restart terminal, navigate back to the directory where the python client was generated and try again.  Cassandra-remote should spew out a verbose usage directive.

That's it; you're done.

If you found this useful, or have feedback, please let me know.  I use gmail (gdusbabek).  I also emit the occasional tweet; just follow gdusbabek. 

If you're interested in learning more about Cassandra, there is an active and helpful IRC channel (#cassandra) on freenode and mailing lists as well.  The wiki also contains useful information for beginners.

id3 for Python

I've been meaning to package up some python code I wrote earlier this year and release it for free as open source software.  Several things held me back from doing this.  The biggest reason, by far, is that I'm still not proficient at python, and feel like I'm exposing myself by putting this code out for the world to see.

But then I remembered that my blog doesn't have a lot of readers anyway.  So no worries there.  And besides, maybe I can garner some constructive criticism to make my python code better.  :)


http://www.dusbabek.org/~garyd/id3_python/

This library is capable of reading most any correct Id3v2.3 or 2.4 tag, some incorrect ones, and then fails gracefully when things get hopeless.  (I run smoke tests on my mp3 collection, which has a lot of nasty debris from the Napster years.) 

It supports unicode, and does a good job handing PIC/APIC tags.  I should also mention it is in production at Tagfriendly.

For those that do come across this, I'm still trying to figure something out.  All the documentation I've come across says that I should structure my directories as such:

MyModule/
    id3/
        __init__.py
        stuff.py
    setup.py
    tests.py

I include id3.py inside the id3/ directory because that's where I think it should go.  But then when I build and test the module, the only way I can access the code is if I import id3.id3, but I wish I only had to import id3.  Clearly, I'm not doing something right.

My solution, and I know this isn't right, is to do away with the id3/ directory altogether and just have id3.py rubbing elbows with setup.py and test.py.  Anyone know what gives?

Well, enough of all that... id3 for Python is available at http://www.dusbabek.org/~garyd/id3_python/.

P.S.  Thanks to my friends on IRC for reminding me about this.

Adventures in Javascript

Note: Blogger doesn't give me a good way to preview the image before it goes live.  If it is too small, my apologies.  I'll try to get that fixed before your news reader pulls the feed.

The Question
Sometimes I think programmers tend to accept things the way they are without really questioning them, especially when it comes to language quirks.  This is something I remember coming across when I was first learning Javascript.  I was reminded of it recently while reading a Javascript book.

Javascript pros will probably quickly recognize what's going on here.

At first I thought the third definition of MyFunc would wipe out the others.  And just to verify that declaring a function in that manner normally makes it into the assumed contexts, I did it with YourFunc.

What gives?

Go ahead, take a guess... 

The Answer
The interpreter evaluates standalone functions before the other expressions.  So it is as if 'function MyFunc...' were written before everything else.  This can be verified by calling MyFunc() in the first line of the script.  The expressions (which include assignments) are then evaluated.  So what appears as the second assignment of MyFunc is really the third, and the third is really first.

Stupid CSS Tricks

I have recently made the committment to wrap my head around CSS.  In other words,  I'm tired of guessing.  Part of this experience will have me documenting my discoveries on this blog.

Here is what I learned today:

Pseudo-classes aren't just for anchors.

That's right.  You're supposed to be able to apply standard pseudo-classes to just about any selector you can come up with.  This means you don't to rely on a) jQuery or b) onmouseover/onmouseout to do the hover-effect work for you, so long as you can do it all in CSS.

Here is some sample code:

<html>
    <head>
        <style type="text/css">
  
        /* a class for an element */
        span.my_hover_class:hover {
            background-color:red;
            cursor:pointer;
        }

        /* attached to nested elements. */
        div > div > span > span:hover {
            background-color:green;
            cursor:pointer;
        }
 
        </style>
    </head>
    <body>
        <span class="my_hover_class">Should hover red (span.my_hover_class)</span>
  
        <div>
            Should not have hover effect.
            <div>
                <span><span>Should hover green (div>div>span>span)<span><span>
            </div>
        </div>
    </body>

</html>

I would like to show the effect here, but the Blogger editor insists on tidying up my pasted HTML.  I uploaded the file to my website.

This example was tested in Firefox and Safari on a Mac.

Hacking the Yahoo! Media Player

I've been using the Yahoo! Media Player to stream Mp3s at Tagfriendly.  Minor glitches aside*, the only real complaints I have is that it isn't skinnable and there is no public API.

About a month ago I started prodding it to see what would make it squeak.  This post documents some of what I've found.

Disclaimer: I skimmed over the YMP terms of service and don't believe I'm breaking any of the rules.  You should also know that YMP is a) beta software, and b) a hosted application.  This means that any code you write or use that relies on specific methods or objects will be brittle and prone to breaking when Yahoo! releases updates.  That said, you're on your own; I didn't make you do anything.

YMP is one of those nifty internet tools you can use by simply embedding a <script> element in your markup.  If the page you've embedded it in contains links to mp3s, or if it links to an XSPF playlist, YMP picks it up and makes those MP3s streamable.  This is powerful if you're a non-technical blogger and want to have an embedded player on your site.

My aim is to leverage all of that, but to take the Yahoo! face off and give it my own.  I use jQuery to manipulate the Tagfriendly DOM, but really, any decent JS toolkit should allow you to get the same results.

The first thing you need to do is know when YMP is finished loading so that you can tell the UI to go away.  Due to the fact that the Javascript you embed includes a bootstrap that downloads other things, you can't count on YMP to be ready when your document is.  This is easily accomplished with a JS timer that polls to check whether or not YAHOO.MediaPlayer.setPlayerViewState is defined.  When it is, call YAHOO.MediaPlayer.setPlayerViewState(YAHOO.mediaplayer.View.DisplayState.HIDDEN); to make the YMP user interface go away.

Now you are in the drivers set to start working with the MediaPlayer object.  Here are some useful methods:

YAHOO.MediaPlayer.getTrackPosition()
     Gets the position offset (in seconds) of the currently playing (or paused) track.

YAHOO.MediaPlayer.getTrackDuration()
     Gets the track duration (in seconds).  YMP appears to grab this from ID3 tags in the mp3, so you can't always count on this piece of data to be there.

YAHOO.MediaPlayer.play()
     Tells the player to play the currently queued song.

YAHOO.MediaPlayer.pause()
     Tells the player to pause the currently playing song.

YAHOO.MediaPlayer.previous()
     Go back to the previous song.

YAHOO.MediaPlayer.controller.EventManager.onNextRequest.fire()
     I found it odd that there was no YAHOO.MediaPlayer.next() method.  This method does what you expect the non-existent next() would.  There are corresponding fire() objects for play, pause and previous as well.

YAHOO.MediaPlayer.getMetaData()   
     Returns an object that describes the currently queued song.  Useful properties there, gathered from ID3 and the XSPF, include 'title' and 'artistName'.  There're more if you care to look.

That's basically it--all you need to subvert the YMP UI and handle things your own way.  This really is a guerrila API hack, as I don't think the YMP designers intended a public API.  You can see the results  on the front page of Tagfriendly.  I went as far as providing a progress bar that gets updated as the song plays.  The Javascript source is freely available too.

In the future I plan on displaying cover art and linking the songs to their Tagfriendly description pages as well as to music stores.



* I suppose not so minor because it really bothers me:  YMP breaks down when you try to use a locally hosted XSPF file or the machine it is hosted on is stuck behind NAT.  Firebug reports that YMP executes a GET with the URI to the XSPF.  The backend of that GET does some data-munging of the XSPF contents.  The results contain a basic JSONified playlist.  The only problem is that if the [development] server that hosts the XSPF is behind NAT, that backend can't fetch anything.

Emotional Investing

Nicole and I take X percent of our income each year and put in various retirement investments. We had a bit left over for 2008 and I've been wanting to invest in wind energy. Specifically, I'm looking for a wind turbine manufacturer who is well-established elsewhere and trying to break into the burgeoning U.S. market. (If you find such a company, and it can be easily traded in the U.S., let me know.)

While researching this I came across PWND. The techie in my wanted to snatch it up just so I could say I have stock in PWND. I mean, that would be l33t, wouldn't it? Anyway, I had to pass it up. What little I know about investing in stocks has taught me to stay away from ETFs.

Back to the research...

Restoring Myth Programs Whilst Maintaining Sanity

My MythTv system has been running continually, more or less, since October 2006. Not too long, but long enough. Long enough, in fact, for cruft to creep in. I discovered this on Saturday when I went to restore data to the replacement for the myth drive affectionately known as "xfs2" that died about a month ago.

That drive had a capacity of 250GB. My myth data is important to me, but not that important. I had it set to back up once a month to another computer in the house via rsync. I didn't pay too close attention to the way I used rsync though--instead of removing the remote files that had been removed locally, they just stayed there. In other words, unneeded data on the backup never went away, even when I deleted shows on the myth system.

So when I went to copy 274GB of data to a drive that would only hold 238GB, things went haywire.

Thankfully, I am a programmer. I am equipped for these kinds of situations.

I needed to figure out which shows missing locally that were present on the backups. These are the files that needed to be restored. After that, I figured it would be gravy if I could remove the orphans that were in the database but not on any file system, be it local or backup.

It turns out that python is incredibly easy to use with MySQL. I created a simple program that would restore from my backup and also give me a list of orphaned programs.

Seems easy, right? The complexity comes when both local and backup storage are strewn across different drives, directories and hosts. I made it simpler by using smbmount to mount the backup system so it appeared more or less local. After that it became a matter of letting the script run* and then cleaning up the orphans with a simple sql statement.

Source code for myth.py is at the bottom of this post.

* This turned out to be tricky. Something on my myth host causes it to intermittently hang when copying files to or from a remote host. It could be my router for all I know. I do know that *any* keystroke received by the myth system causes it to wake up and start accepting traffic again. Meanwhile, the system clock thinks nothing has happened and starts up again at the same tick where it fell asleep--so the clock is off. This problem first started when I decided to upgrade to Ubuntu 8.10 and MythTv 0.21 on the same day. Truly disturbing, I know. I circumvent this by bandwidth-limiting scp to 2Mbit for the transfer. Transfers take longer, but at least they complete. And yes, I am retiring this system in a few weeks. :)

import MySQLdb as db
import os
import sys

existing_video_dirs = ["/xfs1/myth/video", "/xfs2/myth/video"]
# If you use scp, backup_video_dirs and backup_scp_paths need to be maintained
# in parallel. For sure though, backup_video_dirs need to be locally mounted.
backup_video_dirs = ["/mnt/remote_backups/myth_backup/xfs1/myth/video", "/mnt/remote_backups/myth_backup/xfs2/myth/video"]
backup_scp_paths = ["garyd@child:/mnt/xfs3/myth_backup/xfs1/myth/video", "garyd@child:/mnt/xfs3/myth_backup/xfs2/myth/video"]
backup_info_dict = zip(backup_video_dirs, backup_scp_paths)
possible_file_extensions = ["mpg", "nuv"]

# this is the place video gets copied to.
restore_path = "/xfs2/myth/video/"

# connect to the database and get the list of shows.
con = db.connect(host="localhost", user="root", passwd="root", db="mythconverg")
cur = con.cursor()
cur.execute("select chanid, starttime, title from recordedprogram order by chanid, starttime")
rows = cur.fetchall()

# keep track of the number that are there or not.
there = 0
not_there = 0
restored = 0

for row in rows:
    # name of the file is based on row values, plus a file extension.
    fname = "%d_%s" % (row[0], row[1].strftime("%Y%m%d%H%M%S"))
    # see if the file exists in the local myth dirs. If it does, there is no
    # action.
    exists = False
    for dir in existing_video_dirs:
        for ext in possible_file_extensions:
            path = os.path.join(dir, fname + "." + ext)
            if os.path.exists(path):
                exists = True
    if exists:
        there += 1
    else:
        # if the file does not exist locally, look at the backup directories to
        # see if it is there.
        not_there += 1
        backup_exists = False
        backup_at = None # path of backup file.
        backup_scp = None # scp path of backup file.
        #for tup in backup_info_dict:
        for dir, scp in backup_info_dict:
            for ext in possible_file_extensions:
                path = os.path.join(dir, fname + "." + ext)
                if os.path.exists(path):
                    # found a backup! use wild cards so that the video file
                    # and its preview get copied.
                    backup_at = path + "*"
                    backup_scp = scp + "/" + fname + "." + ext + "*"
                    backup_exists = True
        if not backup_exists:
            print "No backup for %s" % (fname)
        else:
            # move the backup to the live system. use cp or scp depending on
            # your preference.
            cmd = "cp %s %s" % (backup_at, restore_path)
            # something on my system is jacked. I need to bandwidth limit scp
            # or else the myth server stalls.
            cmd = "scp -l16000 %s %s" % (backup_scp, restore_path)
            print cmd
            os.popen(cmd)
            restored += 1

# output the stats.
print "there:%d not_there:%d restored:%d" % (there, not_there, restored)

Update: fixed tabbing in code. Should have used pastebin.

The Saddest Story Ever Told

If you depend on a computer like I do, this one will make your innards shrivel up inside you and convince you to become a hermit...

The display unit in my company-owned MacBookPro when belly up on Monday night (thank you, Nvidia), one day before the one-year warranty would expire. (This was actually fortuitous, as my employer did spring for Applecare, but it was lost and never registered to my Macbook. Another story.) I quickly made plans to take the ailing machine to the Apple Store in Salt Lake City on Tuesday. Then I set about deciding on how I would work until I got it back. I am a remote employee; no IT staff to give me a loaner when equipment goes bad.

Here is the lay of the computers in my house:
1. Nicole has a 2.5 year old white Macbook (single core Intel 1.2 GHz?)
2. The kids have a brand new dual core Intel Atom 1.6GHz machine. It has a 1.5TB drive in it and serves mainly as a network storage device.
3. MythTv runs on a very old AMD Athlon 1.3 GHz machine.

Using the Myth server was out of the question. Not only is it too old to be useful in my work, it has reached mission-critical status in our household. We need our TV shows.

Nicole offered, and I breifly considered using her computer. I decided not to because I thought the kids computer might be a bit more powerful (though not a mac) and I didn't want to bother setting up a my dev environment there. You see, I had a plan.

The Plan involved using the kids computer. It was new, though underpowered, but I figured I could make it work fine. Before taking my macbook in I made a copy of the windows virtual machine I sometimes work in. I figured I could install VMWare Player on the kids computer, turn on the VM and I'd be in good shape. Setting up a non-development machine to do development work usually consumes about half a day for me, so I was keen on avoiding this.

But it wasn't meant to be.

The only file system on their computer that had space for the VM was formatted XFS, and I had been having little problems with it that I had mostly chosen to ignore. VMWare started complaining about 30 seconds after bringing the VM up. Then the OS dropped the volume completely. Ouch.

I ran xfs_repair and re-mounted the volume to try again. I'm a hopeful person and besides that, I needed to get some work done. Same exact problem. Ugh.

At this point my thoughts were focused on two things: First, I had a few work tasks that must get accomplished, and I still didn't have a dev environment to do those tasks. Second, I needed to get rid of XFS on that volume. It would not be viable in the short- or long-term.

First things first... I still haven't put the new 250GB hard drive in the myth server to replace the dead one. I decided to stick a USB enclosure on it, format it as ext3 and move the windows VM to it. At least then I could get into the VM without it crashing. The VM ended up performing quite poorly (due to running over USB), but would be ok for an afternoon of work.

Next I had to figure out how to get rid of the crappy XFS partition that was giving me grief. The only problem was that it had about 750GB worth of data that I couldn't just delete. It contained my music backups (expendable), DVD rips (not very expendable, but I could live without them), myth backups (needed for a full recovery if I ever got around to replacing the dead drive in the myth machine) and other assorted backups.

Luckly, I happened to have an extra 1.5TB drive laying around. I plan to put together a new Myth server in a few weeks and have started assembling the new hardware. I briefly tried installing windows XP on that drive (wasn't looking forward to working in a VM), but the aged XP installer didn't like the humongous drove. Oh well. You know the drill by now: USB enclosure, format ext3, and move data off the XFS volume. Ubuntu estimated it would take about 8 hours for the transfer, so I went to bed.

In the morning, the copy looked good, so I removed the USB drive and set it aside. Next I unmounted the XFS partition and reformatted it as ext3. Then I moved my 30 GB VM onto it and fired it up.

It works a lot better. Not only is it stable, but it runs a lot faster (this was expected). I suppose I can work like this for a week until my Macbook comes back.

The next step will be to restore the backup data off the USB drive back onto the backup machine. I'm losing the stomach for hardware maintenance, so I'm going to put it off for a day or two.

NOTE: I've used every model of 15-inch MacBookPro/PowerBook since the Aluminum G4. Each one has needed repairs within the first year (one even shipped with a defective trackpad). My take: always spring for the extended warranty.

iTMS link generator API

I set about looking for an API that utilizes the iTunes Music Store link maker, thinking they would be all over the place (deep links that is, and includes referral program ids). Five minutes of half-hearted googling didn't get me anywhere I needed to be, so I decided to take matters into my own hands. (Note: either I am a crappy googler, or the Apple legal team keeps these things off the air. Seriously, this code must have already been written 10 or 12 times now.)

A bit of poking around revealed that Apple (edgesuite, linkshare, linksynergy, whoever!) uses GET for their search forms. This makes writing a scraper API as complicated as parsing HTML, which is pretty easy in this day and age. I decided to use python, because I'm learning it by forcing myself to use it for all my extra-curricular projects.

I've released the source code under the code section at my website. The license is, uh... liberal. So don't be afraid to use it if you find it useful. It depends on the excellent BeautifulSoup library to do the heavy HTML lifting.

Here is some iTunes deep-linky goodness for you. These are the top 5 most played songs in my library. Buy them:
1. New Slang, by The Shins
2. Valley Winter Song, by Fountains of Wayne
3. Dreams Anymore, by The Magnetic Fields
4. Trust, by Gravenhurst
5. Saint Simon, by The Shins

BTW, I've resurrected Tagfriendly as the mp3 blog aggregator I've been working on, soon to have iTMS referral links. It isn't much, but I'm adding features all the time.

(Tagfriendly was, at one time, an automatic ID3 editing tool I created that mostly worked.)

Mp3 Blog Aggregator: status and some code

My last post was a lamentation about how the current set of mp3 blog aggregators don't do it for me, and at the same time a declaration that I would do something about it.

I've spent my spare moments this week hacking at the problem and it's starting to bear fruit.

The first of it is a simple id3 reader implemented in python. It simply reaches out over the tubes and grabs the id3 information from an mp3 that is hosted on a server somewhere. Nothing too complicated, except that it can be configured to extract any images that might be embedded in the mp3.

Knowledgeable readers might be asking: "why didn't he use one of the three or four existing python id3 libraries?" The answer is this: I planned on creating a blog crawler (mentioned later) and a website for this idea, and would do it all in python. As a warm-up exercise, I figured it would be good to create a simple id3 reader. I had already done it in Java, so it mainly became an act of seeing how the Java idioms I am currently used to translate over into python. (Note: if you bother to download and read the code, please be gentle. It's the first real python I've written. Feedback is appreciated too.)

The crawler is mostly done. It came together more quickly than I thought, although it still has rough edges. It runs a few times a day, notes new blog posts and gathers what information it finds into a database (postgres).

The website is where the work needs to be done now. I have gotten no further than creating a few simple query+display pages that I've been using to view results from the crawler. I've experimented with different ways to present data (entry-centric vs mp3-centric) and still haven't come up with something I like. I've got time though. And the longer I wait, the more useful data I'll have from the crawler.

I'm still using pylons for the website, although I had second thoughts after spending too much time fighting mako and the way it manhandled my nice unicode mp3 tags.

I have yet to tackle the problem of dynamic RSS generation, but I have some good ideas in my head for that.

MP3 Blog Aggregators

I started this post as a "Dear Lazyweb" but decided against it. I'm actually going to do something about this particular problem.

I subscribe to quite a few MP3 blogs. As I told a friend recently, "there is no quenching the thirst for new music." It occupies a fair amount of my internet time, but has lead me to good tunes. And good tunes translates to a happy Gary, so it's time well spent.

One of the biggest problems with MP3 blogs as they exist today is that if you come across something good, there is no easy way to say "find me more like this" without doing all the legwork yourself. Sure, the poster might mention "this sounds like X, Y, or Z," but that is just one persons opinion. I've tried a few mp3 blog aggregators and found a decent one, but there is still a lot noise and I'm not very happy with it. It is just that much better than the competition, which is poor to start with.

Social networking to the rescue. Audioscrobbler has silently been rolling out more APIs over the last 12 months, mostly without anybody noticing. They haven't shut off any of the old services that I currently use (the music section on my website), but they are requiring an API key to use the new services. One of the old services that has been reincarnated in the new is the "find simliar" feature, where an artist or song is supplied and related matches are returned. To be fair, Pandora does a better job of this than Last.fm, but Pandora has no API that I can use.

So the project, and I've already started pounding out code, is to scan the music blogs, figure out how they link to MP3s, grab the ID3 tag and then store that information in a database along with a link back to the original post. Several interesting things could be done with that information:

1. Find me posts (and mp3s) of related artists or songs. That isn't terribly interesting, but takes some of the legwork out of doing it manually.

2. Zeitgeist tracking. The difference between the good music blogs and the less-good blogs is that the good ones go out on their own to find new music, artists and information, rather than recycle what is being hashed on other blogs.

3. Search-based aggregate syndication feeds. Imagine being able to create an RSS/ATOM feed based on aritst or artist-similarity. This feed would aggregate all the posts that you find interesting. For example, you could create a feed that would return all posts mentioning Belle & Sebastian, or posts that contain references to artists similar to Belle & Sebastian. Pre-filtered information like this is a great time saver.

That's about it for now. I don't know if this kind of tool would be very useful for many. But it is fun to hack at and I've had a hard time lately finding recreational programming tasks that engage my passions. Also, this one is my first real venture into python, which is turning out to be quite fun. The website is in pylons, the spider is plain python, and they both communicate with the database using SQLAlchemy.

In the Year 2000...

Dear Lazyweb,

I have a lot of ripped CDs and DVDs; I use MythTv and I have a mac.

And I want to access them all using Frontrow *and* Mythfrontend.

Thanks, Gary

This is one of those problems where the pieces are all 99% there, but not enough people want the solution bad enough (myself included) to put them all together.

Firefly can take a collection of mp3s and serve them up to iTunes/Frontrow. It can even handle video to an extent (must be mov or m4v). That is 50% of the problem, leaving only a way to get at the Myth TV shows.

There are several ways this could probably be handled. To me, the path of least resistance would be this:
1. A vlc setup to transcode and stream the MPEG-2 Myth tv shows into something iTunes can swallow. This is already doable.
2. A plugin for Firefly that will a) query mythtv using either the database or UPnP and then b) stream the video from vlc. There will be a little bit of glue that is needed to map items exported by the database/UPnP to urls for vlc.

I have two weeks off at Christmas. Maybe it's time to get my hands dirty and give something back to the community I've enjoyed for so long.

Sign extension in java

I was talking with a programmer recently about a bit-twiddling problem. The conversation brought to mind things I learned a long time ago when I was new to Java.

One of the very unfun things about java is that there are no unsigned types. This means that if the hi-bit of a byte is on it gets extended when promoting a byte to an int. This applies to 8 bit unsigned values in the range from 128-255 (0x80-0xff).

This leads some some puzzling problems when you're byte munging in Java for the first time. Consider this:


System.out.println(Integer.toBinaryString(128));
> 10000000

System.out.println(Integer.toBinaryString(-128));
> 11111111111111111111111110000000

byte b = 0x80;
System.out.println(Integer.toBinaryString(b));
> 11111111111111111111111110000000

System.out.println(Integer.toBinaryString(0x80));
> 10000000

System.out.println(0x80 == b);
> false

Huh?

At first it seems silly that 0x80 != b when b was explicitly set to 0x80. The problem here is that java promotes b to an integer so it can be compared with 0x80 (which is already an integer, even though your mind wants to treat it as 8 bits). The process of promoting b (which is a negative number as far as a signed byte goes) extends the 1 in the hi-bit.

Another way to explain this is to say that casting 0x80 to a byte converts it from a positive 4-byte integer to a negative 1-byte integer. Casting 0x80 to a byte solves the problem:

System.out.println((byte)0x80 == b);
> true

The java libraries get around this problem by treating all bytes as integers (look at InputStream and OutputStream to get a feeling for this). In that case, it is probably more correct to:

System.out.println(0x80 == (0x000000ff & b));
> true

One more thing to be aware of this that java has an unsigned right shift operator: >>> that always shifts in a zero regardless of sign:

System.out.println(Integer.toBinaryString(b>>1));
> 11111111111111111111111111000000

System.out.println(Integer.toBinaryString(b>>>1));
> 1111111111111111111111111000000

Be aware that this will byte you (har!) when you assume you're shifting an 8-bit value though.

P.S. By popular demand (my wife), my next post will be non-technical.

Bayes Filtering in Javascript

I use google reader for my rss feeds. Something I'm surprised they haven't added yet are "predictive labels" that attempt to classify your feeds depending on how you've trained it.

Arguably, this would only be useful if you have a lot of feeds and wish to weed out noisy posts (like when a technical blogger starts making political posts--ugh!).

Thinking it would be a fun hack I set out to build a naive Bayes classifier in Javascript. It turned out to be easier than I thought.

Some observations:
1. My math is rusty.
2. Fancy mathematical diagrams sometimes don't translate so easily to code.
3. Edge cases still suck.

The fruits of my efforts are included in an iframe below. If you prefer, you can visit the actual page.

The next step would be to inject this into google reader somehow to predictively classify a post. I'm not sure of the best way to do this. Greasemonkey could handle capturing when a post is tagged, but physical storage would be required to store the probability graphs. That puts me squarely in the realm of traditional Firefox plugins. I'll have to do some more investigating.

Fun learning project though.

BEHOLD, MY IFRAME:

Technical Hubris

Programmers must often strike a balance between correct and pragmatic. This is often the difference between do it right and merely get it done. Rarely does the intersection of the two equal one and the other. But if you ask just about any programmer, he/she will almost always prefer a correct approach versus a more-pragmatic-but-less-correct approach when given the choice. Why? That's just the way most programmers are wired. That, and we're kind of arrogant.

The conflict is that software users almost always prefer that software be usable (pragmatic) even if it comes at the expense of being correct.

A case in point: one problem I worked on recently was to generate a unique set of keyboard shortcuts for a given set of buttons. The stipulation is that a shortcut (keystroke) could only be used once and must be included in the text of the button. I created an algorithm that was guaranteed to find a solution if one existed. It did this by recursively finding the characters (shortcut candidates) that were unique to each button over the set of all characters already mapped and all buttons that had not been assigned a shortcut. When the recursive algorithm bottomed out, all possibilities had been exhausted. If there were any buttons left over it was because assigning any shortuct to one of the remaining buttons would require a shortcut to be taken from another button. My algorithm was correct and I was pleased that I had thought of it to boot.

So given two buttons "My Button One" and "My Button Only", it would return the shortcuts as "e" and "l" respectively (first unique character in each sequence). It turns out this isn't as intuitive as returning, say "M" and "B". Ouch! It would have been difficult to make the correctness argument in view of the seemingly nonsensical shortcut characters that were returned.

The solution I settled on was to alter the algorithm to prefer the first character of the first word, and then the second word, etc. when selecting a shortcut. If no shortcut is found it then starts using other characters from the text. Also, I hard-wired a few words to shortcuts (exit -> x, quit -> q, etc.). This algorithm isn't guaranteed to find a solution if one exists (it isn't correct), but it produces shortcuts that make more sense to users.* It is practical instead.

* Geeky/gory details: here is where the new algorithm breaks down. Consider the following buttons "ab" "bc" "cd" "ac". The modified algorithm would produce shortcuts "a" "b" "c" and nothing for the fourth button. But the old algorithm would have come up with "b" "c" "d" and "a". Chances are slim that such a deviant case would ever be encountered in the wild though.