The HTML code that runs your website is most likely designed by a person. People make mistakes and you need to check to see how ‘valid’ your code is. You do want it to show up in every person’s web browser, no matter if they have Internet Explorer, Firefox, Chrome, Camino, Opera, Rockmelt – or any one of dozens of others.

Validate your website code! Did you even know such a thing existed! Michelle Knight walks you though why it is important, how to do it and what to look out for in her article W3C Validation.

Brad Heller joined AboutUs last week and here are his first weeks observations – One and a Half Minds are Better: Learning by Pairing – on the AboutUs Developers weblog.

Many agile development teams keep track of velocity as a measure of their performance.* This is a useful measure, but it shouldn’t be the only one. This past week we realized that another performance metric of an agile team should be how much work it avoids doing.

During a discussion with stakeholders last week, we realized that much of the work in our queue was a temporary fix for a problem we would ultimately solve at a later date. Thanks to the stakeholders keeping their own planning queue in a visible location, the development team was able to see that the proper fix for the problem was scheduled for just a few weeks later. We engaged the primary stakeholder in a discussion about the requirements of the proper fix, and deemed that it would be no more work than the temporary fix. Naturally, we started work on the proper fix, abandoning the temporary one – and eliminated nearly a week’s worth of work!

Velocity is a great metric to track, but perhaps agile teams should also get in the habit of tracking work avoided thanks to successful communication. After all, the best code is no code at all.

*: This, of course, is not the recommended use for velocity, but remains a method of measuring performance.

We have kicked around the idea of having the means of talking about technical matters regarding our website AboutUs.org for a while.

And last week one got rolled out. Granted there are now a whole two posts on it, and you can subscribe to it to be enlightened.

Dev.AboutUs.org

It is beautiful looking and it does the job.

AboutUs developer Karl Matthias will be traveling to O’Reilly Media’s headquarters next Friday to present at a MediaWiki data summit. He’ll be talking about Kiwi, a fast formal parser we’ve developed at AboutUs for parsing MediaWiki style wikitext.

AboutUs.org has its roots in MediaWiki. We began several years ago as a MediaWiki installation with a few custom plugins.

We’ve changed a lot since then. The application has been rewritten in Ruby, and we’ve pulled in a variety of other technologies as we’ve increased our scale and added new functionality. We still do use parts of MediaWiki’s PHP codebase to parse the wiki content contributed by our editors. It’s the best wikitext parser out there, and hard to replace.

We wanted to experiment with alternatives for parsing the wikitext in our database. Wikitext is notoriously difficult to parse. It is a context-sensitive language, and so can’t be parsed with the traditional parser generator tools used to parse most programming languages or grammars. Attempts have been made to formalize wikitext, but so far, none of these efforts are complete.

Ward Cunningham had been experimenting with PEG, a new style of parser which was first
written about in 2004. PEG style parsers combine the tokenizing and parsing stages of the traditional parser generators that have existed since the 1970s. This new parsing philosophy makes formal definitions of context-sensitive grammars like wikitext possible.

Thomas Luce, another developer at AboutUs, went home one weekend and began using a PEG parser generator to create a recursive descent wikitext parser. Within a few days he’d implemented a good chunk of wikitext’s features. I wrote a simple wiki application that uses the parser so we could see the results as it improved. Within a few weeks, with contributions from Karl and Ward, we felt like we were approaching a full wikitext parser implementation – and one that parses at blazing speeds, since it compiles down to C code.

We’ve built an admin-only “parse with Kiwi” feature into AboutUs.org, and may switch over to it completely in the near future. We’ve also open-sourced Kiwi under a BSD-style license so it can be used on other websites. We hope wiki aficionados will find it useful, and that it will be improved by the wiki developer community.

At the very least, Kiwi is cool for demonstrating that parsing wikitext can be done with a formal parser. It was also great to simplify some of our infrastructure in the process of developing Kiwi.

Editor’s note: This is cross-posted at Drastic Code.

We recently re-implemented the site’s search feature here at AboutUs. Behind the scenes the new search uses Sphinx (a super-fast and scalable search engine) and Thinking Sphinx (which allows easy configuration and querying of Sphinx from within Rails).

One thing that makes Sphinx so much faster than alternatives like Solr is that it communicates directly with your database when it is building its search index. This is far faster than talking with your application layer, and indexing the result of your application models’ methods. In our case we’re able to reindex about 40 million records in around 4 hours. Solr used to take days.

There is a major disadvantage to this approach though. Since Sphinx is talking directly to the database, you can’t index the results of method calls on your ruby objects. I’ve discovered though that this is not nearly as limiting as it may sound, since you can index the results of SQL functions. Even better, manipulating and transforming your data with SQL functions as you index it, is virtually free. I haven’t noticed any impact on our index times, after adding many functions to our indexing process.

Here are a couple real examples.

In our app, we deal a lot with domain names, and websites. One feature of our search displays domain names that match your query. Our stakeholders wanted these to be sorted by the importance of the tld, with .com at the top, followed by .org, .gov, etc. Since this sort order is basically arbitrary I replace each tld with an integer, using mySQL’s REPLACE function, store this attribute in Sphinx, and sort on it when querying.

This is what it looks like in Thinking Sphinx’s define_index block:

  define_index do
    # Attributes that allow for fast sorting of domain names by tld, favoring .com's etc.
    # Useful in "exact domain match" search
    replace_str = "LOWER(REPLACE(REPLACE(REPLACE(REPLACE(REPLACE(REPLACE(REPLACE(REPLACE(REPLACE(page_title,'com','1'),'org','2'),'gov','3'),'edu','4'),'net','5'),'co.uk','6'),'de','7'),'co.jp','8'),'info','9'))"
    has replace_str, :as => :page_title_sortable_by_tld, :type => :string

It’s not pretty, but it’s far prettier than trying to do this in ruby, at query time. Now we just add an :order => "page_title_sortable_by_tld ASC" to our calls to Page.search.

Another place we use this is to convert a flag stored in a serialized YAML structure into a boolean attribute that Sphinx can sort on and filter by.

    # Allow favoring in search results based on whether a page is
    # "pro_followed" (i.e. it has its external links followed).
    #
    # Since this data is stored in a serialized YAML structure we need some SQL
    # function fanciness to access this data as a boolean value.  If the below
    # YAML key value appear in the text field LOCATE will return an integer
    # greater than 0.  If not it will return 0.  Coercing this to 1 or 0 (with
    # "> 0") allows sphinx to interpret this as a boolean value.
    join page_data_latest.page_datum
    has 'LOCATE(":follow_external_links: true", `page_data`.`content`) > 0',
      :as => :pro_followed, :type => :boolean

This code searches a field in the database for the some literal YAML text, and stores whether it was found as a true/false attribute in the search index. This is way easier than deserializing the YAML and storing this value in a boolean field in mySQL.

This has proved to be a very useful technique for us, and is far easier and more efficient than denormalizing some of our data to a “search text” column, or doing a bunch of post-querying processing within Ruby on Rails.

In Part I of “How (and Why) to Get Everyone Coding,” AboutUs staffer Ted brought up some situations where there are direct business advantages to having more people with even very simple technical skills. This is the “with enough eyes, all bugs are shallow” saying everyone has heard.

But what about the purely cultural advantages of having non-technical staff coding? Would it still be useful to have staff learn to code, even if no one got real access to the codebase?

He said yes, “because whenever I’ve had a conversation with devs when they get any inkling that I understand what they’re saying, they view me differently. I can see it in the way they treat me. It’s just a different kind of conversation when their work is not so mysterious.”

AboutUs community guru Mark was more skeptical about the ROI of learning to code. He said that for people who are in constant contact with developers, like project and product managers, the benefits are clear. But should startups really be paying a CFO or marketing staff to learn to code? Reactions were mixed on this point, with some people enthusiastic about the idea and others skeptical.

Ward Cunningham helped some of our non-development staff to get more familiar with code. Here’s what he has to say about it:

I surprised the world by showing how collective-ownership can empower a community where anybody can edit. What the world doesn’t yet realize is that wiki is a miniature of the development methods I was creating at wiki’s founding. These methods evolved to become Agile software development.

AboutUs is an agile development shop. Contributing to an agile codebase isn’t much more complicated than contributing to wiki. If its a large codebase then contributing can seem mysterious as it is with Wikipedia. But this complexity comes from sociological effects, nothing inherent in the technology.

I hope that as the world comes to understand large wikis they will also understand that a large computer program, like those we are forced to deal with every day, or even civilization itself, must be made through collective-ownership if it is to remain healthy and last. I hope the staff at AboutUs can serve as an example.

Hopefully this has been a useful example for anyone interested in getting non-technical members of a startup into the code. If you have had an experience with this kind of effort, good or bad, it would be great to get your feedback.

Recently, we came across the closing keynote at PyCon by Antonio Rodriguez. In it, he made what he called a “crazy proposition” that caught our attention. Here’s the main thrust of it:

I think every employee in a web startup— or in fact any company which depends on software in any meaningful way— should learn how to code. From the slickest sales guy to the most obstinate operations guy, from the laziest intern to the most professorial manager, if they don’t have their hands in the code, your startup is much more likely to fail.

AboutUs.org is built on Rails, not Python, but this declaration about culture and code in Web startups was very interesting to us nonetheless. We haven’t gotten everyone in our startup to code, but we certainly have had non-programmers dipping a toe into the codebase. Here’s our experience with such an endeavor, and the reflections of the staff involved.

Tip of the Iceberg

The first serious project that got non-technical staff involved in code was a smaller one, not our main site AboutUs.org. It was headed up by Julia and Ted, who work in operations, finance, and organizational development.

The work mostly involved simple front end development, but some of the facts of life for coders became clear to them quickly. One of these points was the need for good source control. Ward Cunningham took a little time to help them get set up with GitHub.

From then on out, they were largely on their own, using online help sources to work their way through it — coding by Google, basically. All of this has been relatively smooth sailing, and Ted said he thinks it could be the tip of the iceberg for us in terms of getting everyone playing with code.

A Little Knowledge

Ted stressed that it was “definitely” a good idea to have people start small and work on what amounted to a side project. Not working right away on our main product afforded us more room for experimentation.

“The idea of pushing into GitHub is in some ways a scary one,” he said, “for mission critical software, if I pushed something that was a mistake that devs didn’t realize was a mistake, it would still be my fault. That phrase, ‘a little bit of knowledge is a dangerous thing’, it’s true.”

Squashing Small Bugs

So now that some of our staff have tried some front end work and tinkering with code, what about the idea of say, pulling an offsite for non-technical staff to learn to write a simple program?

Ted said, “I think it’s a great idea for a lot of reasons. There are a lot of things that don’t get done because the devs are working on higher priority stuff, and it’s not dangerous. It’s changing wording, for an example. Non-devs can do that and it won’t be a big problem. I love the idea of going even further than that, and learning how the basics of the code work.”

“Even reading commits is a good thing, which some of us have done. We’ve read commits over a period of time, watching what the devs are doing. But if actually we had commit access, even if we didn’t have deploy access, to be able to find small bugs and things that are simple, that would free up quite a bit of the dev time.”

Ted got excited talking about how he and Mark, the AboutUs Community lead, were able to refer to line numbers and other specific features of the codebase when talking with AboutUs developers. Clearly it’s not hard to get some staff interested in learning more about coding.

Read Part II for more, including Ward Cunningham’s thoughts about getting everyone to code.

That’s right, we’re looking for several highly talented folks to to join our Portland, OR or Lahore offices.

Here’s a rundown of the position. If you or anyone you know might be a good fit, email your cover letter and resume to jobs@aboutus.org (be sure to include the job title and city in the subject line).

Agile Rails Developer

Top 10 Rails site (Alexa #715) seeking talent for our small development team. AboutUs.org is a structured wiki of all websites. We run much of our site on AWS and are experimenting with other technologies that scale to support hundreds of thousands of people working together on millions of pages. We’re venture-backed, and our founder’s history of leading startups to successful acquisition helped him win the Entrepreneurship Award for Individual Achievement from the Oregon Entrepreneurs Network last month.

Requirements

We are looking for people to join an elite team. To be successful, you’ll need to have already done the following:

• Deployed web applications
• Wielded advanced programming
  • closures
  • metaprogramming
  • distributed computing
  • service-oriented architecture
• Worked with XP/Agile methods

Read more…

Here at AboutUs, we have hundreds of GBs of data. Most are static, immutable data, keyed by a specific primary key.

With such a use case, key value databases make perfect sense. After looking at various options, we chose to experiment with Tokyo Cabinet & Tokyo Tyrant first, because Tokyo has such a low barrier to entry.

A couple of benefits that we saw immediately from using Tokyo:

• It is fast. Not as fast as in-memory store, but if you have more disk space than memory for storing data — for example, cache data — Tokyo is an attractive choice.
• Tokyo’s performance is affected by number of keys. But it took far more keys to slow it down than we expected.  Tokyo didn’t show any significant slowing even when we pushed 10 million keys in a single database. We were pleasantly surprised with its performance.
• It can handle memcache protocol. This lowers the barrier to entry, making Tokyo attractive for experimentation.

Installation
Installing Tokyo isn’t trivial, but it’s not hard, either. It requires libbz2-dev to be installed first:

# For RPM based
sudo yum install libbz2-dev

# For debian based
sudo apt-get install libbz2-dev

If you want to use Tokyo’s binary protocol — via rufus-tokyo for example — you need to have both Tokyo Cabinet and Tokyo Tyrant installed. We guessed that we might need just the header files, but decided we did  not want to manage those files manually.

First Try — Success!

In our first non-experimental run, we let Tokyo Tyrant handle our page cache. We have been interested in alternative databases for a while, but page cache was the first project that allowed us to try one out.

Each AboutUs domain page is built from at least three individual widgets. Because we intend to create more widgets, building pages on the fly, every render, is clearly inefficient. That’s why we came up with the page cache solution.

Deploying page cache is a smooth process. The size of our casket.tch file has grown to 51 GB, but so far, we’ve had to restart Tokyo Tyrant only once.

Second Try — Fail!

Our next opportunity to try out Tokyo was in making  our domain information easily available to our web application. Our total domain information is  about 160 GB. It’s not a trivial size to handle, and we still want quick access to it, especially since each piece of information is keyed by domain name.

Since our first attempt was a success, we thought Tokyo Tyrant would have no problem with handling our 160 GB opportunity.

Filled with hope, we naively pushed the entire 160GB to Tokyo Tyrant without throttling, using a simple script. As Tokyo grew, approaching 40 GB, it began slow down significantly. After 60GB, it died. We could neither restart the ttserver, nor recover the casket.tch.

In hind sight, we could have:

• Used more than one database.
• Sharded multiple instances of Tokyo based on domain names.

But we didn’t. Instead, we recovered quickly, and used Amazon’s SimpleDB.

Conclusion

We are still very happy with Tokyo’s performance in handling our page cache.

With better design, we could have used it to serve our domain information. But Tokyo is competing against other databases we are currently evaluating, including Amazon Web Services. For us, ease of use is a big deciding factor. But we’re still evaluating.