Tag Archives: search

Safe search on Twitter

Twitter made an announcement today about stopping abusive accounts and hiding low-quality tweets. I think filtering search results in particular is a very good step in the right direction:

We’re also working on ‘safe search’ which removes Tweets that contain potentially sensitive content and Tweets from blocked and muted accounts from search results. While this type of content will be discoverable if you want to find it, it won’t clutter search results any longer.

As I work on Micro.blog, I’ve tried to be mindful of where users can stumble upon posts that they don’t want to see. Replies is a big one, and I’ll be focusing most of my attention on that. But search, trends, and hashtags are also a problem, because they let anyone’s posts bubble up to a much wider audience. I’m launching Micro.blog without them.

PodSearch

This isn’t the first time that David Smith has built something that I kind of wanted to build myself, too. Today he announced a cool side project for searching podcast audio:

You can easily search for a term or keyword and then play the actual audio back to find if it was the section you were thinking about. I even tag the sections with timecoded Overcast links for easy sharing.

I’d love to see David spin this into either a commercial product or set of free tools. He could host more shows, or let podcasters run their shows through PodSearch and export the results. For example, I’d want this for Core Intuition, along with edited transcripts eventually.

iOS 9 search

Federico Viticci has a comprehensive write-up about Apple’s approach to search in iOS 9, including comments from developers. On local app search:

“With local app search, iOS 9 can build an index of content, app features, and activities that users may want to get back to with a search query. Built like a database and already in use by Apple apps such as Mail and Reminders, CoreSpotlight will provide low level access to the index of an iOS device, making it easy to organize and retrieve content users have previously seen, created, or curated.”

I’ve been slowly going through WWDC session videos, but haven’t cracked open the documentation for search yet. Sounds like an important new API for any app that has user documents.

Google Photos

Tim Cook spoke recently about privacy and cloud services:

“You might like these so-called free services, but we don’t think they’re worth having your email or your search history or now even your family photos data-mined and sold off for God knows what advertising purpose.”

I’m going to give you a very cynical translation, which I don’t often do: We are in denial about how much better Google Photos is than what we’re doing at Apple. It is so advanced in terms of search that we won’t be able to match it anytime soon. In fact, we don’t even have anyone working on similar technology at all.

It’s not about being free. I pay Dropbox $20/month to be grandfathered into 2 TB of storage so that I can put all of my photos and documents there. Dropbox is rock solid and worth it.

Like Marco and others, I have tried to avoid Google services. I don’t use Gmail. I hate advertising. But the idea of being able to quickly search my photos by content without even tagging or organizing them was too compelling to not try. So I’ve uploaded over 10,000 photos so far to Google Photos. It is really good. (I’m going to finish uploading all my photos and give it a few months before making a final judgement on the search vs. privacy trade-off.)

Some of the random searches that work out of the box to filter my photos: “beach”, “trains”, “New York City”, “Oregon and 2013”, “road trip”, “party”, “basketball”, “Christmas tree”. I never saw a demo or tutorial for how to use Google Photos; I just type stuff in and it mostly works, discovering photos and events. And on top of that, there’s also the automatic stories and collages, which is something we always wanted to build for Sunlit.

My family photos are the most important files I have on my computer, and I very rarely share any photos of my kids publicly. But ironically I’m willing to overlook some of the privacy concerns around this exactly because the photos are so valuable to me. I want multiple copies in the cloud, and I want the power of search that Google has built.

Twitter in 2 years

Marco predicts that third-party Twitter apps will lose half of their users within the next 2 years:

“We won’t even be angry at Twitter — we’ll move to the official apps voluntarily, and we’ll look back on all third-party clients like we look back on Tweetie, vanity link shorteners, and third-party image hosts today: as relics of a quickly abandoned past before we all started using Twitter’s better, newer features.”

During WWDC this year, Buzz Andersen gave a great talk at a small venue outside the conference. With the hindsight of several years, he talked about building Birdfeed, the challenges of competing with Tweetie, with his own struggle at perfection, and many more insights on the rise and fall of third-party Twitter apps.

It left me with a lot to think about, and I loved the old stories, screenshots, and related nostalgia. But in closing out the questions & answers, one statement in particular struck me as a nail in the coffin for third-party developers: Buzz revealed that even he now uses the official Twitter app.

Winding down my Twitter apps

I’m the guest on this week’s Mac Power Users podcast. In addition to workflow and apps I use, the discussion went off the rails a little into the Twitter app ecosystem, especially the fact that I no longer post to Twitter yet still have apps like Tweet Library, Watermark, and the Tweet Marker API that depend on Twitter. For the last 2 years this has been an odd decision on my part; I want to do the right thing for my customers, but I’m increasingly frustrated with life as a third-party Twitter developer.

Last week, Twitter announced that they’ve expanded their search index to include the full history of tweets going back to 2006. I was thrilled by this upgrade to the Twitter service. That the search was so limited for so long was the primary reason I built Tweet Library and Watermark to begin with. Unfortunately, this functionality is only for the official Twitter apps. It will not be made available to third-party developers.

It’s time for me to wind down development on my Twitter-related apps. I’ll continue to sell Tweet Library through the end of 2014, then remove it from the App Store. Watermark will also shut down at that time. Because all the tweets stored in Watermark are public tweets (by design it never supported DMs or protected accounts), I will attempt to make the entire Watermark database archive of millions of tweets available publicly. Existing customers can also sync tweets and collections to Dropbox for personal archiving.

Published collections from Tweet Library or Watermark will be maintained indefinitely. No URLs will break, ever. Updating published collections will also continue to work for anyone who already owns Tweet Library.

I will also continue to host the Tweet Marker API, but starting in January I will be more strict about requiring developers to pay for the service. Many developers have been paying for API access for a year (thank you!), but others have missed or ignored my requests to move to a paid plan. It’s not fair to the Twitter developers who have been paying for Tweet Marker access if some continue get the API for free.

Many friends have told me over the years that I have too many products. But letting any one product go is not easy. There’s an implicit promise when shipping software that the developer should maintain and improve it for customers. Stopping development on these apps is the right decision and possibly long overdue, but it’s still difficult. What gives me hope is that it will let me focus on new projects currently in development, which I couldn’t be more excited about.

Searchpath tweaks

I rolled out a few improvements to Searchpath last week. The popover now “fades in” when first shown, using a CSS animation. Scrolling in the background page content is also now disabled while the popover is shown, for WebKit browsers. This prevents the page from bouncing around as you scroll to the end of the search results. And the mobile interface is better, finally including a close link for when you want to get back to the web site.

I’ve also fixed an issue with how often Searchpath looks for new posts. It should be much faster to pick up on changes now, especially for blogs that update every day.

You can try Searchpath for free or subscribe for just $8/month.

Announcing Searchpath

Today I’m happy to announce my new web app: Searchpath. It’s search for your web site or blog with an innovative “popover” UI. Simple, fast. With better control of your search results, and no need to link to Google or show ads to your readers.

There are so many sites out there that don’t have search, or have very poor search. I wanted to build something that makes setting up a great search box on your site absolutely trivial.

Searchpath knows about blogs — it finds the text on your page to index, and can automatically strip out redundant titles from posts for clearer search results. It tracks popular search terms. It gives you more control, by letting you exclude archive pages and customize the font and links with CSS.

I’m running it now on this blog. Check it out in the sidebar, or visit searchpath.io to sign up and try it on your own site.

New product coming soon

Last year I started working on something new. It was going to be the first product after I refocused Riverfold around a new mission statement: apps to keep and remember what matters. It starts by solving a very basic problem, but the long-term scope is very big.

It’s for anyone with a web site or blog. It’s incredibly simple to set up. It’s a web app with a free trial that works like magic, requiring no registration.

Finally I’ve been able to dust off the project, give it a new name, and get it ready to ship. Launching tomorrow, February 14th.

Indexing the hidden web

There has been some good commentary on Sergey Brin’s interview with the Guardian. It’s probably best summarized in John Gruber’s comment:

The assumption here is that the only way to search is through Google, and that the “open Internet” is only what Google can index and sell ads against.

This resonates with me because I think Google has put enough ads on the internet. It’s impossible to take anything Google says at face value if they talk of “open” but their intentions say “ads”. But here’s the thing: Brin is actually right. There is great data hidden behind apps that should be indexable.

In the race to win the App Store, we forgot about the web. Think about Instagram. Millions of photos are being shared that are inaccessible via a search engine. These photos can’t be found again and aren’t discoverable. When you search the web, how does it make sense that public photos on Flickr show up, but public photos on Instagram do not?

We shouldn’t have to choose between Apple’s closed systems and Google’s ad-driven business. I want to talk about improving the web without automatically being pro-Google. This tweet from John Marstall made me realize it:

I’d prefer to separate “indexable” from “indexable by Google.” Yes, they’re mostly the same for now. Might not always be.

We need competition in web search. More than Bing. A new search engine is the number 1 item from Paul Graham’s frighteningly ambitious ideas:

The best ideas are just on the right side of impossible. I don’t know if this one is possible, but there are signs it might be. Making a new search engine means competing with Google, and recently I’ve noticed some cracks in their fortress.

Of course it’s crazy, but so was the 1st search breakthrough, lightning-fast AltaVista, and so was the 2nd major innovation, Google itself. It’s time for a search engine that isn’t all about ads. It’s time for search that understands apps and embraces data from web services as much as it does from web sites. It’s time for the 3rd act in search.

ParseKit (and Clipstart search)

The first couple versions of “Clipstart”:http://www.riverfold.com/software/clipstart/ had a very basic search feature. You could enter keywords and it would search filenames, tags, and video titles. You could also enter special terms such as tags=christmas or imported=today, but you couldn’t mix and match different terms together.

When I started working on a more advanced search parser, I realized that I was about to write a bunch of code that surely someone had already generalized and shared with the world. Tada! “ParseKit”:http://parsekit.com/ by Todd Ditchendorf is that framework.

Clipstart 1.3 now supports these kind of searches:

christmas and (@julian or @kids)

also…

(uploaded=no and flagged=yes) or (date=2010 and @vacation)

I use ParseKit’s tokenizer to take these apart and then I translate to SQL myself for SQLite. New in 1.3, Clipstart also allows saving any search as a “smart tag” for quick access. I’m very happy with how well it’s working.

Why not use “NSPredicate”:http://developer.apple.com/mac/library/documentation/cocoa/reference/Foundation/Classes/NSPredicate_Class/Reference/NSPredicate.html and friends? I wanted more control over the parser, for example for the @kids shorthand for tags. Eventually I’ll have a more traditional NSPredicateEditor-like UI for managing searches, but I find that text input is a much quicker way to find things in my video library.