CS and the City

This week I’ve been taking a look at the recently announced pubsubhubbub by Brad Fitzpatrick and Brett Slatkin of Google. The duo proposed and implemented a protocol for implementing near-realtime notifications on top of RSS and Atom. The protocol describes three roles: A publisher, a subscriber, and a hub. The hub basically acts as an intermediary, receiving subscription requests from subscribers and forwarding update notifications from publishers to subscribers.

One of the first things I noticed about the protocol is that subscribers are required to have an internet accessible URL for validating subscription registration and receiving notification pings. This is not an issue for the Google Readers and FriendFeeds of the world, but this does leave desktop RSS readers out of the party.

Also interesting to note is that there’s nothing that requires the hub to be a separate entity from the publisher. In fact, it could be very desirable for the publish to own the subscription hub. Besides removing one notification roundtrip from the protocol, it would also give publishers more control over how often to ping users on updates. Nothing in the protocol requires that a notification be sent every time, so it would be possible to only notify a subset of users in real time (perhaps the ones that pay), and others on a regular basis.

Depending on how deep your RSS Trivia knowledge goes, this might sound awfully close to the rssCloud element, but Brett points out that the key differentiator here is PSHB’s “fat pings“, that is, the entire updated content is sent as the ping to the user.

To reduce latency and polling, PSHB supports persistent HTTP connections from hubs to publishers, but it could use FriendFeed’s SUP protocol to detect updates as well.

Though solving slightly different problems, it’s interesting to compare the SUP’s and PSHB’s stance on polling. SUP obviously relies heavily on polling, despite drastically reducing the amount required. While PSHB has strong opinions against. Polling is certainly less error prone, in addition to being less efficient. For example, how does PSHB handle dropped pings to subscribers? I admittedly haven’t dug too deep, but I assume a reasonable amount of state must be maintained in the hub to handle these cases smoothly.

Ultimately the most valuable contribution of the entire project might be the two outspoken Google employees behind it. Already they are seeing some adoption. The pubsubhubbub demo at Real-Time CrunchUp announced launched FeedBurner support and showed prototypes of Blogger and Reader support. Having evangelists inside the company puts early adoption in other Google products much more likely, which in turn will give the standard much more credibility.

Leopard (OSX 10.5.4 but it looks like earlier versions were affected) seems to have some sort of bug that causes easy_install or setup.py install to crap out with an error like this:

No eggs found in /var/folders/Ev/Evg2gG5nFyKz5eSgYhyKuk+++TI/-Tmp-/easy_install-Qm7ReE/simplejson-1.9.2/egg-dist-tmp-9qR_OW (setup script problem?)

I ran into this on a number of different packages: simplejson (above), appscript, and others. The fix seems to be this seemingly unrelated bad boy here (discovered here). The fix? This command:

sudo easy_install http://www.jaraco.com/ASP/eggs/setuptools-0.6c8_svn15fix.egg

Ugh.

Part 1 – The Past and Present

The Download Window or Download Manager is the part of the interface that provides the user with information about the file or files currently downloading.  This sort of interface is common in any application that involves downloading (or uploading) multiple files that take more than a few seconds to transmit. The download window you’re most familiar with is almost certainly the one included in your web browser.  A typical day’s worth of browsing usually includes the downloading of a some document or data.

Despite the download view’s frequent use, it seems to be one of the most neglected parts of the browser.  It has improved since early consumer browsers, but the download progress view still manages to trip up in many cases. Admittedly, it’s hardly the sexiest element of a web browser, but it certainly deserves more attention than it receives.

Join me on a quick trip in the way back machine to see just how little the download progress view has changed and where I believe it should be headed.

The Stone Age

Things were brutish in the early days of the world wide web.  Each downloaded file had its own window, and while most users weren’t downloading files any bigger than a few megabytes, dial-up connections guaranteed a transfer would take long enough that a user would inevitably ask, “How much longer?”

If you had multiple downloads going at the same time, not only would they take longer to complete, but your windows taskbar would be overflowing with download progress windows.

This was the way browsers worked back then.  Both Internet Explorer and Netscape Navigator offered this workflow.  Even the first few versions of the Official BitTorrent client worked this way.

I put this style of functionality in the Stone Age, but it doesn’t mean that it is a thing of the past.  Internet Explore 7, the newest version of the browser STILL believes that one download warrants one window.

The Bronze Age: One window to rule them all

The concept of a download manager had been around in the stone age in the form of third-party plug-ins.  Enterprising individuals quickly realized that a central place to track all those long-running downloads would be a feature Joe Surfer would pay for.  It wasn’t until my first encounter with Firefox in 2004, however, that I found my built-in first download manager.

The concept of a download manager is straight forward: It is a place to manage all of your downloads.  If you were to have multiple downloads running at the same time, you could see the status of all in one window.  No more desktop and taskbar clutter (sort of).

I mentioned that Firefox was my first browser to include a download manager, but I am unsure if it was the first of all browsers.  By the time I switched to OS X in 2005, Safari already sported one.

While the aggregation of all the downloads went a long way to simplifying the user experience, it was not yet the perfect user interaction.  Download managers still required significant input from the user to manage:  Download folders needed to be specified continually, history cleaned, windows lost, found again, and eventually closed.

The Iron Age: The road to enlightenment

This is where we stand today: On our way, but certainly not there yet. Many of the major players has given the standard download progress view its own twist.  Let’s quickly touch on some of the progress each has made.

Firefox 3

Firefox 3 adds the ability to opt out of opening the download manager window completely when downloading a file.  Instead, it shows you a simple summary of progress in the status bar. Alternatively, you can set the window to close automatically when the download is complete. And it allows you to set a default download location.

Safari 3

Safari automatically downloads files to the user’s Downloads folder.  If the user wants to download to a different directory, they can through the context menu.

In OS X 10.5, the Downloads folder is configured as a Stack by default on the desktop.  The stack acts as both a download history (by showing the files in cronological order, as well as a notification of completion (as it bounces when a file is added). This is the only truely compelling use of OSX’s stacks I’ve come across.

Unfortunately, with all this functionality, Safari still brings up the download manager.  Settings can clear history automatically when a download completes but it can’t close the window (Yes, a third party plug-in can add this feature, but that’s beside the point)

Opera

Opera opens a separate tab with download status and history. I was impressed with this functionality as I’m not an active Opera user, but I’m concerned that there’s some possibility of the tab getting lost or confused with other web browser tabs.  Seems to be a global concept that should be introduced into the interface some other way.  I’m on the fence here

Internet Explorer

Despite leaving IE back in the stone age, it still has a feature that Safari, Firefox, and most BitTorrent applications lack:  IE hides the downloads until they’re complete. If a file is on your desktop or in your download folder, you can be confident it’s ready for your use.  Other browsers tend to leave their half-downloaded unmentionables in plain view.

The Age of Satisfaction: Attainable?

Here are a few take-aways from the features mentioned above:

Convention over configuration – This has been emphasized heavily as a software development paradigm, but it applies to user interface as well: The browser should act on logic defaults without asking, but provide the user with the flexibility to change their mind.  A perfect example of this is Safari’s downloading to the Download folder automatically, unless the user choose a different location through the context menu.

Get out of the user’s way – Do not open additional windows unless specifically asked to (starting the download is not that request). Provide simple, passive feedback that a download has started and that it has finished. Only showing files that have completed downloading is passive feedback that they are ready to be used.

That does it for our summary of the past and present of the download manager in your friend neighbourhood web browser. Next week, I’ll post up a break down of the workflow from a user standpoint and use some of these take-aways to put together a few mocks of my ideal UI.  They’re not completed yet, so please feel free to weigh in and influence the result.

I’ve been helping the local guys at VendAsta on their secret project for the last few months.  It’s been a great experience working for a Saskatoon-based startup using a lot of really forward facing technologies. It’s also been very cool to work in Saskatoon’s homegrown tech park: Innovation Place.

Innovation Place has all the facilities you’d expect, including a most excellent cafe with daily specials.  To help me make the most out of the cafe’s offerings, I hacked together a simple Google Gadget to do the heavy lifting.  Fellow IP-ers feel free to add it to your own page.

I’ve done a few simple gadgets in the past, but this was a good refresher into the current state of gadgets. For those interested, here are a few thoughts on the space:

• The original Google Gadget API is being depreciated in favour of the gadget.* API specified in OpenSocial.  You can take a look at the new gadget API here.  I haven’t done enough digging to figuring out any substantial architectural differences, but please leave a comment if you come across one.
• The new Gadget API is not supported on the current version of iGoogle.  The iGoogle developer sandbox does, however.
• The old documentation used to recommend using Google Pages to store gadget assets.  However, Google’s recent move to kill off Pages in favour of Sites has removed this option for prospective developers.  Perhaps Google App Engine could serve a similar purpose?
  • Related thought: Build an GAE application solely to host other people’s gadgets.  Static gadgets are simple, but some thought would be needed on the tools required to host dynamic gadgets.  Also a way to pass GAE hosting costs on to gadget developers.  I’ll have to give this some more though.
• OpenSocial gadgets on iGoogle have a lot of potential. 

I hadn’t even finished writing my post announcing my new love of Python when I stumbled into one of its skeleton-filled closets: XML.

The Python core libraries include six different methods for parsing and creating XML, none of which feel particularly Pythonic (here I am, three weeks into developing with Python and already I’m calling out core libraries as not being Python-y enough).  I missed the low overhead methods I had used in other languages. Particularly for parsing XML, PHP’s simplexml is hard to beat, and for building, it’s hands-down Ruby’s XML Builder.  Off I went, hunting for Python ports.

Warning: The following is a tangent.

This may be an unfair statement, but I get the impression that there’s a slight “Not built here” bias in the Python world.  Python has a substantial number of best of breed functionality, both core and third-party, but my initial impression is that they’re a little reluctant to adopt solutions championed by other languages.  Example: Where’s the Python equivlent to CPAN or gems?

Here’s my point:  I found Python ports, but they lacked in the qualities I loved about Python.  Maturity and Active Development.

First on my list was simplexml for PHP.  It allows the developer to access attributes and text through variable and list combination.  To get similar functionality in Python, I found handyxml. Handyxml allows equally brief tree traversal and iteration of multiple items.  Unfortunately, it hasn’t been updated since early 2004 and a lot of the dependencies have moved or are gone completely.  As such, it required some modifications just to get it into a functional state.  Not ideal.

The other functionality I missed was XML Builder in Ruby.  XML Builder takes full advantage of blocks in Ruby to allow nesting of xml element creation that makes the structure of the resulting document blindingly obvious in the code. This is in stark contrast to the Java-esque series of createNode, appendNode that Python (and Java and Objective-C) love. I managed to dig up a recent port by Jonas Galvez of XML Builder for Python.  He took advantage of the upcoming ‘with’ statement in Python 2.6 to achieve the same effect.  Though it had some problems handling unicode characters (remind me to submit a patch to github) and the documentation is minimal, I was able to get it up and running very quickly.  Better.

I know from my digging in recent weeks that there’s been some talk about refactoring Python’s urllib/urllib2 code for Python 3 to simplify the module and remove duplication.  I sincerly hope the XML libraries fall underneath the same knife, and that the solutions from Ruby and PHP are considered for a graph.

As mentioned earlier, I’ve been working on a medium-scale project written in Python. It’s the first time I’ve used python beyond a few lines in a script. After a few weeks of working in Python I returned to an application I had been developing for the Mac in Objective-C over the weekend. It wasn’t until I made the switch back to a “traditional” static typing language that the sheer beauty of Python struck in full force.

It’s really not fair to pick on Objective-C here. A number of languages could sit in its place. Nonetheless, after working with Python for only two weeks, coding in Objective-C felt like being stuck in the middle of a traffic jam where all the other cars are driven by lobotomized chimps.

Every one of my intentions had to be slowly and laboriously explained in great detail to the computer lest I cause a massive digital pile up. To avoid, I was required to take several trips to Apple’s mediocre documentation before I assembled enough square brackets to build the Eiffel Tower. Even once the appropriate method was called, trying to break down the over-abstracted object return types to get the simple data I wanted resulted in so many code-compile-crash loops it hurt.

I wondered how much of my life I had already wasted prefixing all the class names with “NS”, or how many more times I would have to chase down some archaic memory error only to find that I had forgot to put a @ before my string.

Don’t get me started on strings in Objective-C either. I don’t know how the language designers at Apple can respect themselves when it takes almost 50 characters to do a replace (Actual Method Signature: stringByReplacingOccurrencesOfString:withString:), oh and you have to give up backwards compatibility with 10.4 if you want to use it.

This only showed me just how much I loved Python. String slicing. Unicode strings. Dictionaries and Lists everywhere. Generator Functions. List comprehension and filters. Easy to understand/parse syntax. Lots of Third-party modules. With simple APIs. That are open-source. And are actively maintained. Mature. And well documented.

I found myself writing far less code and accomplishing much more. In fact, I distinctly remember feeling excited about just how much functionality I had accomplished in such little time (Django is partly to blame for this).

I has used Ruby a reasonable amount before, but I didn’t fall head over heals largely because of the lack of maturity. The maturity of the documentation of both the core library and third-party modules is one of the most important features of a language for me, and something I missed tremendously coming from a Java background. In fact, I’ve re-written the ruby script that powered my Pitchfork Reviews gadget to be Python. It took me about 30 minutes.

I’m in love.

This week I’ve started some development work that has placed me completely out of my comfort zone: Coding on Python on Windows XP.  I’ve played with Python on a few small one off apps on my own, but this is the first large scale project I’ve taken on with Python as its base.  So although I’m familiar enough with the language to get started, it’s the development environment for writing the code that has me at a loss.

I’ve been trying a myriad of different editors and IDEs trying to find the best tool with the most flexibility.  Unfortunately, the two seem to be opposed.  I believe this is because my short experience with each tool means the “best” is largely the easiest to use.  Easiest to use means more often than not, flexibility is sacrificed for focus.  I’ve been through Crimson Editor, JuffEd (which I am keeping around for simple text editing), GVIM, jEdit, Wing, PyScripter, and Eclipse with PyDev.

The other guys on my team are fans of VIM, but I haven’t been able to motivate myself to overcome its learning curve.  Not that I have anything against purist coding, quite the opposite in fact.  I find the most efficient way to write HTML, CSS, and JavaScript is with a text editor.  Most of my PHP work has been done in TextMate (on OSX), but also on a smaller scale.  I know I need a few more tools to assist in team collaboration, and I’d prefer having an IDE wrangle them all together.

So I came at it from another angle: What features am I looking for that the programs I’ve seen so far just didn’t manage to satisfy?  Well here’s a quick list:

• See a projects worth of files (in the form of a Project/File browser)
• Multiple files open at the same time, managed with tabs
• Hotkey/button code execution
• Run code through the debugger (not a requisite, but would be useful)
• Integration with source control
• Pull code documentation from files in project
• “Jump to definition” navigation
• Indentation management (already a peev)

During my research, I found a lot of neat validation-style applications (for my reference: epydoc, pychecker, pylint, pyflake, pep8, doctest, pyunit, nose, thumb.py, figleaf, coverage.py, pycover).  Being able to tie some of these in to a validation process on build or, dare I say inline, would be inter-spectacular.

Does such a beast exist? I’ve spent enough time with Eclipse, I should be able to bludgeon it into doing what I want, but that’s hardly the elegant solution I was hoping for.  Come on magical blogosphere, do your thing!

Well, to be fair, it’s pretty slow, and some other people have ran into some instability, but the iPhone SDK works just fine on my old iBook G4 allowing me to put off that inevitable new hardware purchase just a little bit longer.  Thanks to Gordon Turner for the tip. Looks like this WWDC ticket just paid for itself!

How-tos are located here and here (do both, the second is a little bit of clean up detail omitted from the first).

Douglas Crockford, Yahoo!’s JavaScript Architect, gives a great introduction to “the world’s most misunderstood programming language.”

It’s a four part-er so I’ve included the links for your benefit rather than a mass embedding of the videos:
The JavaScript Programming Language (1 of 4)
The JavaScript Programming Language (2 of 4)
The JavaScript Programming Language (3 of 4)
The JavaScript Programming Language (4 of 4)
Presenation slides (Warning: ZIP link)

I highly recommend you watch it all. There is much in the way of background explaining just why features of JavaScript ended up the way they are, and a lot more to help Java veterans like myself create a proper mental model for JavaScript development.

Doug has also presented “Theory of the DOM” and “Advanced JavaScript” neither of which I have had the chance to take in yet. You can check them all out on Yahoo! Video.

Side note: I don’t like Yahoo! Video nearly as much as its competitors. The base player stalled out on me and wouldn’t let me advance to my previous place in the video when I restarted. I suspect it’s linearly buffering the content in the background and that was the issue, but there’s absolutely no indication that I’ve run out of buffered content or how much buffering has been completed.

My Team Lead asked me to do a little digging into Service Oriented Architecture before my co-op term wrapped up. I got a copy of Service-Oriented Archiecture by Thomas Erl from the library and starting flipping through it. Some CTO/CIOs don’t understand their developers disdain for Enterprise buzzwords, but I can sum this one up for them in one sentence I discovered in Erl’s text (top of page 37):

“The application of service-orientation principles to processing logic results in standardized service-oriented processing logic.”

At first glance, it reads like an impressive sales pitch. But at second, you realize there is absolutely no content whatsoever in that sentence. None. It is completely devoid of any useful information. Coders find beauty in simplicity. Enterprisiness is anti-beautiful.