So it was really gratifying to see Macworld recognize Tweet Marker with an Editors' Choice Award for 2011. Thank you Macworld for seeing Tweet Marker as an important part of the Twitter experience.

And thanks to all the Twitter app developers who have supported Tweet Marker in their apps. We are up to 9 supporting apps across 5 platforms — Mac, iPhone, iPad, Chrome, and Android — with more on the way. I've opened up the API to over 40 different clients in various stages of research or testing.

Tweet Marker is a little unique among most of the other Eddy winners this year in that it's still completely free. I won't see a sales spike following the announcement. If you've been enjoying the service, consider picking up a copy of my iPad app Tweet Library, or donating directly on the site.

But at $200/month for a dedicated database, there's a very real cost with PostgreSQL on Heroku. For a service like Tweet Marker that is currently free, I wanted to save money on hosting and simplify things down to a single database. I might have preferred to streamline to PostgreSQL-only, since it's rock solid and Heroku's tools are excellent for managing backups and moving data around, but Resque (which Tweet Marker uses to process background work) is built on Redis. If I wanted to streamline to a single database, it had to be to Redis.

Tweet Marker as essentially an OAuth-wrapped key/value store anyway, so why burden the app with a full relational database? The handful of extra tables I had for tracking users and sessions could also be migrated to Redis. And even though it's a memory database at heart, it is persistent, so I can use it for real data.

I conducted the transition in stages:

• First, I was using Redis for stats. Because Redis has an atomic increment command, it's trivial to use it to keep counters on various metrics: total number of reads/writes per hour, day, or month, broken down by different Twitter clients. This was my first use of Redis outside Resque tasks and it performed great.
• Next, I moved the marker data. I wrote a script to dump all the data from PostgreSQL to Redis, and at the same time modified Tweet Marker to write data to both locations when requests came in. I let that run for about 2 days to confirm that the data was identical in both places before shutting PostgreSQL down and routing reads to Redis only.
• Finally, I finished moving some additional tables to Redis so that I could completely shutdown PostgreSQL. Again, simple is better.

All of this was using Heroku's Redis To Go add-on. I love it because there's no configuration; click a couple buttons and you have a server. But as soon as I was running full time on Redis To Go, I started to be concerned that I had given up a great amount of control over my data. While Redis To Go conveniently offers daily backup snapshots, it doesn't appear to have any commands for restoring data or otherwise getting access to the server and Redis files. And though I could probably set up a separate Redis To Go instance and manually send it a slaveof command to turn it into a slave, not being able to edit the config file meant that if the server was rebooted I would lose the slave until it could be reset. Not good.

I also expected I would quickly outgrow the smaller Redis To Go plans, eventually forcing the monthly pricing up to the same level as I was paying for PostgreSQL, but with less control. I had to run my own database server instead. And because Heroku is all on Amazon EC2, it also had to be on EC2 for the best performance between web server and database.

I went to work provisioning an EC2 instance. Initially it ran as a Redis slave to the master on Redis To Go. After I felt comfortable with EC2 and had automated backups in place, I added a separate instance on Amazon for the master and switched the app over to it. The cost for 2 "micro" instances is about the same as I was paying for Redis To Go, but with much more available memory, hourly backups to S3, and access to the Redis aof log. And my total monthly hosting costs are about half what they were before.

I'm hoping to keep this setup for at least the rest of the year and well into next. As the Tweet Marker user base grows, I'll bump up to higher-capacity EC2 instances. Heroku continues to be really great, so I don't expect to change anything there.

The longer story is that SXSW is always changing. I started this blog on the first day of SXSW 2002, when the conference was just a few rooms in a single hallway, and I've seen it grow to more than a few venues spread across downtown Austin and even farther out. That's okay. There's a place for small events, as SXSW once was, and there's a place for the event where the blogging, design, social network, and software folks can meet in one place.

Take the Mac and iOS development world. This year alone has NSConference, Voices That Matter, 360|iDev, 360|MacDev, CocoaConf, MacTech, Çingleton, and SecondConf. These attract developers from all over the country, but most people can attend one at most, and many events are regional conferences at heart.

We need WWDC as the single place. No matter how great the smaller conferences are, WWDC is the big one, the one you don't miss. And so it is with SXSW.

I believe SXSW 2012 is going to be fascinating to watch. A couple years past when everyone already thought it was too big, 2012 could see real turnover. Some previous attendees will skip it, and many new people will speak for the first time. I want to see that conference, to find out what its themes and focus will be, and hopefully our talk can be part of it.

Here are a few tips to keep in mind while using apps that support Tweet Marker:

Turn it on! First things first: enable Tweet Marker in both apps on every device you plan to use Twitter on. MacStories has a good summary of this (with screenshots) for their coverage of Twitterrific and again for Tweetbot.

What syncs? Only the main timeline and mentions sync between the apps. While working with these developers, we made a decision to change how Twitter lists sync after Twitterrific had already shipped. So while both apps think they are syncing lists, the format on the server is not compatible between the two. A future version should fix this.

How to switch apps? Most apps take the approach of saving the last-read position when the app is quit or sent to the background, and restoring it again when it is launched or brought to the front. You may need to change your default habits a little to get the most accurate behavior. When you know you are going to switch apps, just close your active Twitter client first before opening the next one. It may also help to wait for the app to download new tweets before scrolling or interacting with it.

First, the good news: the service is up and fast. Twitterrific is sending a lot of traffic, even with the preference off by default, but the server app is architected in such a way that things are zipping along nicely. Performance is especially important on the iPhone, where a Twitter client might be saving state on quit and any lag would be pretty obvious. Non-trivial server work, such as hitting the Twitter API, is done in the background using Resque, and the queue size there is usually small too.

There have been two distinct problems, though. Each was a good opportunity to improve the system.

On the first day, the background queue stalled and the requests piled up, so much so that my Redis instance ran out of memory and clients started receiving errors. I upgraded Redis, restarted the app, and bumped up the number of processes to quickly catch up on the backlog. Although it hasn't resulted in errors since, it has delayed some requests on 2 more occasions after launch. I also adjusted how I'm collecting statistics to further improve performance.

More recently, I've noticed some sporadic SSL errors when Tweet Marker tries to verify the user account via Twitter. I added more wait-and-retry logic, and I'm keeping an eye on this to kick the background script if necessary. There have been no errors in the last couple days.

I still think it was the right decision to host on Heroku. Although I am manually monitoring the server for general health (need to automate this badly), everything is less work than I would do with a colocated box or VPS. The biggest cost is that at the last minute I upgraded to Heroku's dedicated PostgreSQL database. I'm still evaluating that, but I would rather launch with too much horsepower and scale it down later, than not enough.

Again, thanks for your support. It is a little nerve-racking because I know that other people are depending on Tweet Marker being up and performing well. This is a different experience for me than working at a larger company with a dedicated system administrator, and also different than server backends for my Riverfold products, where it is only my own customers who will be disappointed if I fail.

At some point in a recent update to Safari or Mac OS X, Miis started appearing blank in Wii Transfer. These are rendered in an offscreen window using Flash, then saved as thumbnails in Wii Transfer with the option to export a JPEG. The fix — for reasons I'm still not clear on — was to switch to using NSBitmapImageRep's cacheDisplayInRect instead of initWithFocusedViewRect. Needless to say, customers were frustrated that this was no longer working, and I'm sorry I didn't take care of the problem more quickly.

I've been thinking a lot lately about the future of Wii Transfer and its companion @wii Twitter account. This little Mac app is overdue for a rename and refocused feature set.

More apps are coming, too. Craig Hockenberry, from a Macworld article by Dan Moren:

"The real value is having something that works well across the entire Twitter ecosystem. It's our belief that Tweet Marker will become more useful as more clients support it."

I struggled with how to build, price, and launch Tweet Marker. The first developers to use it are taking a risk, so I felt I had to remove the friction of supporting the service by making the API free. To cover hosting costs, I plan to later make an optional paid subscription available for developers who want more than the basics.

But we're not there yet, and some people have asked how they can support Tweet Marker directly. As an experiment, I've put up a donation button over on the Tweet Marker site. Think of it as a small investment in the service, a bootstrap to get things off the ground and remove the stress of scaling.

It is not, however, a substitute for supporting client developers. Please pick up a copy of Twitterrific, and when my app Tweet Library is available with sync I hope you'll consider that as well. Thanks!

I talked about some of the good and bad of being a Twitter developer on the ATX Web Show last week. There have been a string of changes that cause developers to scramble: turning off basic auth, discouraging mainstream clients, disabling DMs for xAuth. With each step, Twitter loses a little goodwill, and that's demonstrated in the tweets I collected over the xAuth change.

Even as Twitter passes 1 million registered apps, there's a risk that some developers will stick with the platform as users only, putting their apps in maintenance mode. In May, Kiwi developer Isaiah stopped development of his Mac Twitter app:

"I'm just going to take a break from Kiwi for a while. It's still for sale. I still support it. I'll still fix bugs when they crop up. But adding new features and playing catch up with the other guys/gals is off the table."

Maybe because I don't have to depend on Tweet Library sales, I tend to more stubbornly ignore what is good business sense. There's so much I still want to do. As I wrote in my previous take on the state of the platform: "I'm a little discouraged, but not enough to stop."

I think that's doubly true today. More annoyed, but also more determined to plug holes in the platform, from archiving to syncing. I couldn't be more excited about the developers who are building in Tweet Marker support.

And there's always a chance, a feeling that something big is just around the corner. That if I don't add that one feature, or open up that new API, I'll miss the tipping point that makes Tweet Library really take off.

But it almost fits when you give it a name like "Pro" (or iPad Retina, or whatever). This isn't a replacement for the current iPad; it's another layer to the product lineup. And like the awkwardly-named iPod Photo from 2004, I bet the iPad Pro is meant to be temporary. It's a way to sell a high-end, over-priced and over-pixeled iPad before the technology is cheap enough for the masses. A year or two from now, the Retina Display will be available in all iPads, and the "Pro" name will fade away, just like iPod Photo did when all iPods got a color screen.

"Each of these new features tickle the persistent 'push' connection and trigger some action on the device. The short-form state may be transmitted immediately and set on any connected device within moments. Document syncing is likely to trigger a negotiation process to compare the state on any one device with The Truth stored on Apple servers and replace the document on the device with the latest revision — this has the advantage of limiting the window between syncing where conflicts are most likely to occur."

Sync speed matters. The first note sharing server I built for VitalSource years ago assumed a lot of offline time, and despite my blogging in 2007 that it was "magic", in practice it could take 5-10 minutes before all your computers got their act together to get a set of highlights completely synced. With that kind of lag, note edits might happen on a client in the meantime, so we remembered conflicts everywhere and had a UI for resolving them.

Too complicated. The new system, recently rolled out in Bookshelf for iPhone and iPad, syncs so much more efficiently and quickly that conflicts don't need the same emphasis. We can throw away a bunch of code and simplify the user interface.

I've yet to do anything with iCloud except read the release notes and sit through a couple WWDC sessions, but we're going to have a fantastic platform if it can deliver the same speed and reliability of Push Notifications. Guy's post is the first I've seen to connect the dots, capturing how well-positioned Apple is to use this plumbing for all sorts of stuff.