id3 for Python

I've been meaning to package up some python code I wrote earlier this year and release it for free as open source software.  Several things held me back from doing this.  The biggest reason, by far, is that I'm still not proficient at python, and feel like I'm exposing myself by putting this code out for the world to see.

But then I remembered that my blog doesn't have a lot of readers anyway.  So no worries there.  And besides, maybe I can garner some constructive criticism to make my python code better.  :)


http://www.dusbabek.org/~garyd/id3_python/

This library is capable of reading most any correct Id3v2.3 or 2.4 tag, some incorrect ones, and then fails gracefully when things get hopeless.  (I run smoke tests on my mp3 collection, which has a lot of nasty debris from the Napster years.) 

It supports unicode, and does a good job handing PIC/APIC tags.  I should also mention it is in production at Tagfriendly.

For those that do come across this, I'm still trying to figure something out.  All the documentation I've come across says that I should structure my directories as such:

MyModule/
    id3/
        __init__.py
        stuff.py
    setup.py
    tests.py

I include id3.py inside the id3/ directory because that's where I think it should go.  But then when I build and test the module, the only way I can access the code is if I import id3.id3, but I wish I only had to import id3.  Clearly, I'm not doing something right.

My solution, and I know this isn't right, is to do away with the id3/ directory altogether and just have id3.py rubbing elbows with setup.py and test.py.  Anyone know what gives?

Well, enough of all that... id3 for Python is available at http://www.dusbabek.org/~garyd/id3_python/.

P.S.  Thanks to my friends on IRC for reminding me about this.

Tagfriendly Updates, 1 Jun 2009

I've been quietly hacking away, albeit slowly, on the next iteration of my Tagfriendly project.  The back end is in a state I am happy with, so this version is all about making the front end suck less.

You can sample the work as it progresses at http://www.tagfriendly.com/tf2. This is truly alpha-quality work, lacking much in the way of CSS, but I link here because I know a few people follow this blog and are interested in it.  That, and since I'm not actively working on the current version, I feel the need to show something for my efforts.  :)  I push updated code live about once or twice a week.

I use more AJAX this time around, including fancy transitions and assembling the page in chunks.  I'm not sure if this is the right approach yet or not; I'll see how far it gets me.

In a few days I'll start work on the new player, now that I've discovered there is a published API for the Yahoo! Media Player (that is very hard to find via google).

And Now, A Completely Unrelated Gripe:

I offcially hate eBay.  I sell one or two items on eBay per year, so things are usually in a different place every time I need to sell something.  I can handle that, but two things tonight really bugged me.

1.  I could not find a way to sell my stuff without signing up for and agreeing to automatic payment of seller fees.  This needless hoop was in my face and prevented me from doing what I needed to do.

2.  I have an unopened seasons 1-5 boxed set of 7th Heaven that I want to get rid of.**  Well, in its wisdom, eBay has decided to cap shipping on DVDs at $3.  In other words, I couldn't charge more than $3 if I wanted to.  But this boxed set weighs about 5 pounds and will easily cost me $6-7 ship.  I realize what they're trying to do, which is crack down on S+H abuse, but at what cost?  They know the exact item I'm selling because they had me enter the SKU.  They should know how much it weights and what it would cost to ship through various methods.  Total fail, eBay; it's craigslist from here on out.

**I really thought Nicole liked the show and it would make an excellent Christmas gift.  It turns out she only watches the show on Myth because she can skip the commercials and watch it at 150% speed.  Total fail, Gary.

Hacking the Yahoo! Media Player

I've been using the Yahoo! Media Player to stream Mp3s at Tagfriendly.  Minor glitches aside*, the only real complaints I have is that it isn't skinnable and there is no public API.

About a month ago I started prodding it to see what would make it squeak.  This post documents some of what I've found.

Disclaimer: I skimmed over the YMP terms of service and don't believe I'm breaking any of the rules.  You should also know that YMP is a) beta software, and b) a hosted application.  This means that any code you write or use that relies on specific methods or objects will be brittle and prone to breaking when Yahoo! releases updates.  That said, you're on your own; I didn't make you do anything.

YMP is one of those nifty internet tools you can use by simply embedding a <script> element in your markup.  If the page you've embedded it in contains links to mp3s, or if it links to an XSPF playlist, YMP picks it up and makes those MP3s streamable.  This is powerful if you're a non-technical blogger and want to have an embedded player on your site.

My aim is to leverage all of that, but to take the Yahoo! face off and give it my own.  I use jQuery to manipulate the Tagfriendly DOM, but really, any decent JS toolkit should allow you to get the same results.

The first thing you need to do is know when YMP is finished loading so that you can tell the UI to go away.  Due to the fact that the Javascript you embed includes a bootstrap that downloads other things, you can't count on YMP to be ready when your document is.  This is easily accomplished with a JS timer that polls to check whether or not YAHOO.MediaPlayer.setPlayerViewState is defined.  When it is, call YAHOO.MediaPlayer.setPlayerViewState(YAHOO.mediaplayer.View.DisplayState.HIDDEN); to make the YMP user interface go away.

Now you are in the drivers set to start working with the MediaPlayer object.  Here are some useful methods:

YAHOO.MediaPlayer.getTrackPosition()
     Gets the position offset (in seconds) of the currently playing (or paused) track.

YAHOO.MediaPlayer.getTrackDuration()
     Gets the track duration (in seconds).  YMP appears to grab this from ID3 tags in the mp3, so you can't always count on this piece of data to be there.

YAHOO.MediaPlayer.play()
     Tells the player to play the currently queued song.

YAHOO.MediaPlayer.pause()
     Tells the player to pause the currently playing song.

YAHOO.MediaPlayer.previous()
     Go back to the previous song.

YAHOO.MediaPlayer.controller.EventManager.onNextRequest.fire()
     I found it odd that there was no YAHOO.MediaPlayer.next() method.  This method does what you expect the non-existent next() would.  There are corresponding fire() objects for play, pause and previous as well.

YAHOO.MediaPlayer.getMetaData()   
     Returns an object that describes the currently queued song.  Useful properties there, gathered from ID3 and the XSPF, include 'title' and 'artistName'.  There're more if you care to look.

That's basically it--all you need to subvert the YMP UI and handle things your own way.  This really is a guerrila API hack, as I don't think the YMP designers intended a public API.  You can see the results  on the front page of Tagfriendly.  I went as far as providing a progress bar that gets updated as the song plays.  The Javascript source is freely available too.

In the future I plan on displaying cover art and linking the songs to their Tagfriendly description pages as well as to music stores.



* I suppose not so minor because it really bothers me:  YMP breaks down when you try to use a locally hosted XSPF file or the machine it is hosted on is stuck behind NAT.  Firebug reports that YMP executes a GET with the URI to the XSPF.  The backend of that GET does some data-munging of the XSPF contents.  The results contain a basic JSONified playlist.  The only problem is that if the [development] server that hosts the XSPF is behind NAT, that backend can't fetch anything.

Tagfriendly Update

I am going to tie off version 0.1 since I've accomplished the meager goals I set for myself in January with regard to Tagfriendly. Here is what they were:

1. Track music blogs.
2. Full-text search.
3. Dynamic RSS feeds (turned on last night).
4. Learn python (still a work-in-progress).
   

I've added other features along the way, but those are the basics that I wanted. Here are some things I've been thinking about for the next iteration (roughly one month):

1. More Last.fm (social) integration. Lots of ideas here. For example:
• What if when you scanned the recent MP3s, you could know which ones your friends like?
  
• What if you could create an RSS feed that would generate entries related to your top 10 (20 or 30) songs or bands? (this was suggested by a friend.)
  
• What if you could scrobble love/hate directly from the website?
  
• Lots of cool things here, and they won't be too hard to implement.
• The problem with Last.fm is that using them has a detrimental impact on Tagfriendly site performance (their services are S-L-O-W). I've used a lot of caching (artist information mostly), but that only gets me so far.
  
2. Simple identity management. This will enable features like "show me all the new stuff since the last time I was here." I'll start with just a cookie. That has limitations, so I'll eventually push for simple email registration.
3. Flash based in-browser mp3 player to stream the songs from the blogs. The ethics of this are probably dubious, but I have noticed that many of the bloggers deep link to mp3s stored on other sites. I guess we're all in this together. Hopefully I can find something off the shelf that does this.
4. I need to clean up the back-end scripts that do the crawling and generate the static pages.
5. Better tests.
6. More blogs. I'm up to 30 and crawling+indexing get done in about 3-4 minutes. I'd like to scale up to 100 blogs for this iteration and see how things are handled.
   

That's about it for a month. I average about an hour a day of Tagfriendly development, so this is about the right amount of work for a month. (Honestly, I could probably spend an entire month on testing, but how sexy is that?)

Full-text Indexing on the Cheap

After Christmas I decided I wanted to build an mp3 blog aggregator that would be all my own. I've decided to chronicle the process here on my blog.

I have been a big fan of Lucene for years, although I now prefer Solr (it's still Lucene) for new projects. The main reason I've stuck to Lucene is that most projects I've worked on have been Java-related and limited hardware resources haven't been an issue.

All that changed with the latest incarnation of Tagfriendly. I develop it on a linux VM, which has all the hardware resources of my workstation (more than enough to handle Lucene), but I deploy Tagfriendly to a VPS where my VM is one of many that may be sharing the same physical hardware. To put it simply: I am hardware-constrained, currently stick with 128MB RAM and 5GB of disk space. For example, consider this: apt-get update craps out if I'm running lighttpd+paster. Yeah, it's like that.

While I can grow to a bigger VPS if I need to, I figured it would be good to build Tagfriendly to be lean on resources. Bottom line: I couldn't consider using Lucene as the search engine. So I started to look at other options.

The first two I considered were tsearch2 and sphinx.

tsearch2 kind of surpised me. The last time I looked into it was several years ago when it was a postgres contrib module--not part of the standard release. Even though the idea of having a full-text search structured like a SQL query was alluring, I didn't consider tsearch2 for long. I knew I wanted to be able throw just about anything into the index and rely on a "type" field to restrict results to particular data (e.g., mp3s or blog entries).

That left me with sphinx. I came across it while I was googling for Lucene alternatives. It has python bindings, is written in C, and claimed to be very lightweight. Seemed like a good fit. One downside I noticed is that once a sphinx index is created and initially filled, it can't be added to. That means I would have to generated the index from scratch every time I need to update it. Ouch. (Maybe this isn't true in newer releases.) I kept Sphinx on the list, but kept looking.

A day or two later, before I had written any indexing code, a friend pointed me to Xapian, which has a lot of the characteristics of Sphinx, except that Xapian is more mature (and can be updated). I settled on using Xappy which is a python wrapper around the basic python bindings provided by Xapian. Xappy simplifies the fielded aspect of managing a Xapian index (makes it feel more like Lucene). After playing with both the standard python Xapian bindings and xappy, I recommend using Xappy for simple indexing projects. The learning curve for the traditional Xapian API is pretty steep.

A few hours after finding Xappy (spread out over a few nights of work), I managed to produce a simple index page in pylons and an indexing daemon for Tagfriendly. The index only contains about 1500 documents at this point, so searching is still quite snappy. We'll see how it goes as the document count grows.

With an index in place, I am free to move on to some of the really interesting features and ideas I want to explore with Tagfriendly. For example, I am excited to implement a feature where users can create RSS feeds based on custom search criteria (e.g. includes Morrissey, but not The Smiths).

iTMS link generator API

I set about looking for an API that utilizes the iTunes Music Store link maker, thinking they would be all over the place (deep links that is, and includes referral program ids). Five minutes of half-hearted googling didn't get me anywhere I needed to be, so I decided to take matters into my own hands. (Note: either I am a crappy googler, or the Apple legal team keeps these things off the air. Seriously, this code must have already been written 10 or 12 times now.)

A bit of poking around revealed that Apple (edgesuite, linkshare, linksynergy, whoever!) uses GET for their search forms. This makes writing a scraper API as complicated as parsing HTML, which is pretty easy in this day and age. I decided to use python, because I'm learning it by forcing myself to use it for all my extra-curricular projects.

I've released the source code under the code section at my website. The license is, uh... liberal. So don't be afraid to use it if you find it useful. It depends on the excellent BeautifulSoup library to do the heavy HTML lifting.

Here is some iTunes deep-linky goodness for you. These are the top 5 most played songs in my library. Buy them:
1. New Slang, by The Shins
2. Valley Winter Song, by Fountains of Wayne
3. Dreams Anymore, by The Magnetic Fields
4. Trust, by Gravenhurst
5. Saint Simon, by The Shins

BTW, I've resurrected Tagfriendly as the mp3 blog aggregator I've been working on, soon to have iTMS referral links. It isn't much, but I'm adding features all the time.

(Tagfriendly was, at one time, an automatic ID3 editing tool I created that mostly worked.)

Mp3 Blog Aggregator: status and some code

My last post was a lamentation about how the current set of mp3 blog aggregators don't do it for me, and at the same time a declaration that I would do something about it.

I've spent my spare moments this week hacking at the problem and it's starting to bear fruit.

The first of it is a simple id3 reader implemented in python. It simply reaches out over the tubes and grabs the id3 information from an mp3 that is hosted on a server somewhere. Nothing too complicated, except that it can be configured to extract any images that might be embedded in the mp3.

Knowledgeable readers might be asking: "why didn't he use one of the three or four existing python id3 libraries?" The answer is this: I planned on creating a blog crawler (mentioned later) and a website for this idea, and would do it all in python. As a warm-up exercise, I figured it would be good to create a simple id3 reader. I had already done it in Java, so it mainly became an act of seeing how the Java idioms I am currently used to translate over into python. (Note: if you bother to download and read the code, please be gentle. It's the first real python I've written. Feedback is appreciated too.)

The crawler is mostly done. It came together more quickly than I thought, although it still has rough edges. It runs a few times a day, notes new blog posts and gathers what information it finds into a database (postgres).

The website is where the work needs to be done now. I have gotten no further than creating a few simple query+display pages that I've been using to view results from the crawler. I've experimented with different ways to present data (entry-centric vs mp3-centric) and still haven't come up with something I like. I've got time though. And the longer I wait, the more useful data I'll have from the crawler.

I'm still using pylons for the website, although I had second thoughts after spending too much time fighting mako and the way it manhandled my nice unicode mp3 tags.

I have yet to tackle the problem of dynamic RSS generation, but I have some good ideas in my head for that.

MP3 Blog Aggregators

I started this post as a "Dear Lazyweb" but decided against it. I'm actually going to do something about this particular problem.

I subscribe to quite a few MP3 blogs. As I told a friend recently, "there is no quenching the thirst for new music." It occupies a fair amount of my internet time, but has lead me to good tunes. And good tunes translates to a happy Gary, so it's time well spent.

One of the biggest problems with MP3 blogs as they exist today is that if you come across something good, there is no easy way to say "find me more like this" without doing all the legwork yourself. Sure, the poster might mention "this sounds like X, Y, or Z," but that is just one persons opinion. I've tried a few mp3 blog aggregators and found a decent one, but there is still a lot noise and I'm not very happy with it. It is just that much better than the competition, which is poor to start with.

Social networking to the rescue. Audioscrobbler has silently been rolling out more APIs over the last 12 months, mostly without anybody noticing. They haven't shut off any of the old services that I currently use (the music section on my website), but they are requiring an API key to use the new services. One of the old services that has been reincarnated in the new is the "find simliar" feature, where an artist or song is supplied and related matches are returned. To be fair, Pandora does a better job of this than Last.fm, but Pandora has no API that I can use.

So the project, and I've already started pounding out code, is to scan the music blogs, figure out how they link to MP3s, grab the ID3 tag and then store that information in a database along with a link back to the original post. Several interesting things could be done with that information:

1. Find me posts (and mp3s) of related artists or songs. That isn't terribly interesting, but takes some of the legwork out of doing it manually.

2. Zeitgeist tracking. The difference between the good music blogs and the less-good blogs is that the good ones go out on their own to find new music, artists and information, rather than recycle what is being hashed on other blogs.

3. Search-based aggregate syndication feeds. Imagine being able to create an RSS/ATOM feed based on aritst or artist-similarity. This feed would aggregate all the posts that you find interesting. For example, you could create a feed that would return all posts mentioning Belle & Sebastian, or posts that contain references to artists similar to Belle & Sebastian. Pre-filtered information like this is a great time saver.

That's about it for now. I don't know if this kind of tool would be very useful for many. But it is fun to hack at and I've had a hard time lately finding recreational programming tasks that engage my passions. Also, this one is my first real venture into python, which is turning out to be quite fun. The website is in pylons, the spider is plain python, and they both communicate with the database using SQLAlchemy.