| Contributing to Open Source projects |
[Mar. 20th, 2010|06:12 pm]
Brad Fitzpatrick
|
Prior to joining Google I always joked that Google was the black hole that swallowed up open source programmers. I'd see awesome, productive hackers join Google and then hear little to nothing from them afterwards. When I joined I decided I'd solve this mystery and post about it but it's been over 2.5 years and I've been busy and somewhat forgot. Fortunately a discussion at work last week reminded me of this again, and a bunch of us got to talking about the phenomenon.
Just as there are rarely absolutes in anything, there are no absolutes about open source programmers' activities after joining Google. The main reasons for them sometimes disappearing, as far as I can tell, are:
- Many open source programmers are just programmers. They like working on fun, hard problems, whether on open source or otherwise.
- They're busy. Google seems to suck everybody's free time, and then some. It's not that Google is forcing them to work all the time, but they are anyway because there are so many cool things that can be done. I often joke that I have seven 20% projects.
- The Google development environment is so nice. The source control, build system, code review tools, debuggers, profilers, submit queues, continuous builds, test bots, documentation, and all associated machinery and processes are incredibly well done. It's very easy to hack on anything, anywhere and submit patches to anybody, and notably: to find who or what list to submit patches to. Generally submitting a patch is the best way to even start a discussion about a feature, showing that you're serious, even if your patch is wrong.
Personally, my increased involvement with Google side-projects and decreased involvement with public open source projects is a bit of all three of those bullets.
Notably, though, I want to discuss the last bullet.
It's pretty difficult to figure out how to contribute in the open source community. Given some package on your system or some tarball you downloaded, it's not always obvious what the right process is for that community to get patches upstream. It's often a research project just to find the upstream version control system, or bug tracker, or the mailing list to send patches to. CONTRIBUTING files in tarballs, if present at all, are often out of date.
When you're used to this, perhaps it's not so bad, but inside a company with a very consistent and easy-to-hack-hack-hack environment, this can be daunting. I'm not just talking about Google here. I'm sure most companies have more internal consistency in tools & processes than the collective open source community.
My request:
So here's my request to the open source community: make a webpage for your project that summarizes your community's development resources & process. And then link the hell out of it. Link it from all over your project's documentation. Make sure you have a CONTRIBUTING file, but don't put the current information in the file.... it'll just get stale. Instead, put your contributing documentation URL in your CONTRIBUTING file. Tools and processes change, but tarballs get old, and distros are rarely bleeding edge.
Good examples of people doing this already (from a quick search) include Django, Mono, and MySQL.
If your project doesn't already do this, as most of mine haven't, or haven't well enough, I made a website to make this easy:
Contributing: http://contributing.appspot.com/
Anybody can (and should!) use that for their project to create a project page with a stable URL listing their project's resources and quick summary of the project's development workflow. Where's your source, bug tracker, code review tool, style guide, mailing list, etc?
I've been creating project pages for all projects I'd started in the past, and making sure to update all their docs and websites with links to the Contributing page.
Here are some of mine:
http://contributing.appspot.com/memcached
http://contributing.appspot.com/perlbal
http://contributing.appspot.com/sgnodemapper
http://contributing.appspot.com/contributing
http://contributing.appspot.com/djabberd
....
Still creating them, but afterwards I hope to be able to filter more of my mailing list subscriptions and not feel guilty about people having out-of-date information and emailing me directly.
From now on I will never either a) fail to document the contribution process for a new project I start, or b) document that sending me patches directly is the answer. That may be true for a bit, but projects often change hands, and stale documentation sucks. |
|
|
| Google Profiles has XFN now |
[Jun. 24th, 2009|01:38 pm]
Brad Fitzpatrick
|
Google Profiles just launched a new feature that's too dorky and
obscure to warrant an official "Google blog" blog post, so the product
manager on it said, "Brad, you're dorky... you should post it. You do
Social Graph API stuff. The right people would read your blog,
right?" (roughly)
So sure, I'll blawg it here.
Google Profiles now
have XFN rel="me"
attributes on links. Again. (It had them briefly for awhile but it
was done grossly so they were removed...)
Why is this important? rel="me" links are the
glue of your social identity online. They tie together all your sites
& accounts, letting
other sites know where to find you. (Of course, if you don't want
to be found, or have different personas: don't make links between
them!). But if you're reading this post you already know all this, so
I'll shut up.
How does it work in Google Profiles now? While I don't work
directly on Profiles, I sit near them and like to voice opinions on
things. So here's the new design, which you can blame me for parts of
if you hate it:
- assume users don't care about
rel="me" and it's super
dorky.
- do the best possible right thing by default, but let dorks
override it.
- assume users will use products in ways you didn't imagine (aka
"wrong")
- assume users will add Profiles links to their favorite websites,
bands, friends, etc., not just "their" pages on the web.
- hide the
rel="me" choice by default when adding a link
- show the
rel="me" choice if they go back and press "edit" on it
- track two new bits per-link:
- does the user care about
rel="me"? (i.e. are they dorky?)
- if so, does the user want this link to be
rel="me"?
- when rendering the Profiles page HTML, consider those two bits:
- if the dork bit is on, use the value of the second bit (whether
they chose
rel="me" on this link)
- if the bit is off, just guess. But guess somewhat
conservatively. We can adjust these heuristics over time (a lot of which are based on sgnodemapper), as
most the links will be in do-not-care mode.
So, my dorky friends, you can now fix the rel="me"
state on your links by going
to the
editor and pressing "Edit" on the links and checking their state.
Be sure to hit "Save" at the bottom.
Enjoy.
(And keep in mind that the real utility of all this comes later.
Consider yourself a dorky earlier adopter.) |
|
|
| One Year. |
[Aug. 19th, 2008|11:25 pm]
Brad Fitzpatrick
|
Today was my 365th day at Google.
Tomorrow begins year two. |
|
|
| Perl on App Engine |
[Jul. 22nd, 2008|08:38 pm]
Brad Fitzpatrick
|
Fellow Perl hackers,
I'm happy to announce that the Google App Engine team has given me permission to talk about a 20% project inside Google to to add Perl support to App Engine. To be clear: I'm not a member of the App Engine team and the App Engine team is not promising to add Perl support. They're just saying that I (along with other Perl hackers here at Google) are now allowed to work on this 20% project of ours out in the open where other Perl hackers can help us out, should you be so inclined.
As background, I've been writing Perl code for almost 15 years now and quite fond of the language. (I'm "bradfitz" on CPAN.) Here at Google, though, it's not one of our big languages so I don't get to write as much Perl as I used to. I'd still like to run my personal web apps on App Engine, though, and I'd like to write them in Perl. And I'm definitely not alone, looking at how many people have starred the wishlist bug. Some of you have already started talking about it. We'd like to join the discussion, and start hacking out in the public.
In the process we can build the start of an open source App Engine server clone that's suitable for many purposes: initially just for regression testing & local development (like the "dev_appserver" that comes with the App Engine Python SDK), but perhaps in the future (once Hypertable/Hbase/etc are ready) a full stack to give to ISPs to let them run App Engine apps on their own.
Before I get into my proposed roadmap, let me describe what's publicly known about the App Engine architecture. In a nutshell, it looks like this:
The App runs in a multi-layer hardened environment, one layer of which will need to be a hardened Perl interpreter.
Basically, we need a hardened Perl runtime which can:
- open & read files
- NOT write files
- NOT open sockets
- NOT fork
- NOT do any other system functionality that's not strictly needed for a web app
Basically we need a Perl interpreter that's very tame and isn't allowed to do anything other than read web requests and write out responses. Any privileged operations (like Datastore access, fetching URLs, etc) need to be done via a trusted XS Perl module (the "apiproxy") that takes a service request parameter and returns a service response. The request and response are both encoded as Protocol Buffers, which were recently open sourced by Google.
Perl on App Engine then would involve the following steps (in no particular order):
- Hardened Perl Interpreter: basically, we'll be statically linking in a hardened, customized libperl to a C++ application, disabling all Perl dynamic loading. Only vetted, security-audited XS modules will be allowed. Only safe Perl opcodes will be allowed. (No sockets, no ioctl, no fork, etc, etc.) To get a preview for what this'll feel like restriction-wise, check out the newly written Sys::Protect which Artur and I wrote this evening and will be continuing to develop for people's dev environments (not production).
- Protocol Buffers for Perl: we need support for Protocol Buffers for Perl. I've started on this project internally and will open source the code soon, once I have a few free minutes.
- Server: we need to write an App Engine server for testing, local development, and potentially production deployment. (Replace Bigtable with MySQL, Hypertable, Hbase, Couch DB, etc.)
- Libraries: Perl client libraries for Datastore, URLFetch, etc services. Including docs.
Not included is the Google-internal side of things, gluing the hardened Perl interpreter into the GAE world. That needs to be done by a Googler and not open source.
If you'd like to discuss this and/or help out, join the perl-appengine mailing list. We'll be submitting code to the appengine-perl project on Google Code hosting. For more information about this, see the Perl-on-AppEngine FAQ.
Brad & the other Perl Googlers |
|
|
| Proud |
[Feb. 29th, 2008|12:53 am]
Brad Fitzpatrick
|
Between Google and San Francisco, I don't know which I'm more proud of that when I search for "sf" I get pictures of zombie attacks:
 |
|
|
| libxml security problem |
[Jan. 11th, 2008|09:48 am]
Brad Fitzpatrick
|
I found a security problem in libxml. And by "found" I mean "ran into and debugged a bit".
From http://mail.gnome.org/archives/xml/2008-January/msg00036.html : * From: Daniel Veillard
* Subject: [xml] Security flaw affecting all previous libxml2 releases
* Date: Fri, 11 Jan 2008 07:05:01 -0500
Unfortunately, a security flaw was found (originally by Brad Fitzpatrick
from Google) and affecting all previous releases of libxml2 when parsing
XML. Two specially crafted broken UTF-8 sequences when occuring at the
wrong place lead the parser to go into an infinite loop. Very annoying,
as this lead to a relatively easy Denial of Service attack, the good part
being that this is very unlikely to happen just by error, and to protect
the community we won't release the way to reproduce this.
But all users are strongly invited to upgrade their libxml2 versions to
2.6.31 [1], or apply the patch [2] (or a derivative for 2.5 or 2.4 branches)
to their version. Most OS vendors shipping libxml2 should have updates
by now or very soon, if needed check your update stream, it is referenced
as CVE-2007-6284 .
Sorry for the inconvenience,
Daniel
[1] ftp://xmlsoft.org/libxml/libxml2-2.6.31.tar.gz
[2] http://veillard.com/libxml2.patch
So, yeah... go update your libxml if you process untrusted XML and don't want your CPUs spinning.
(Amusingly, this might be the only publicly visible thing so far that I've worked on at Google...) |
|
|
| Misc |
[Dec. 23rd, 2007|01:31 pm]
Brad Fitzpatrick
|
I've been in Moscow for the past week for this Google Code Day [video], giving a talk on social graph / interop stuff ("Открытие Социального Графа"). It was a public talk, so I should be able to post the slides, but I'll wait and confirm before I do.
Went to SUP's holiday party too. [some pics] But I guess those pics don't really highlight the ridiculousness of the venue. All the nightclubs here are pretty extravagantly ridiculous. But there was no George Clooney or Gwenth Paltrow in a cage. (love that article: you don't often see the phrase, "We OWN you, bitches!" in a printed newspaper with your breakfast, hotel restaurant windows looking out to St. Basil's in the Kremlin...)
I just got a call from the hotel receptionist saying that my driver was here. My driver? What? Oh, my flight was originally today, so probably my return taxi to the airport, but I'm returning now instead on Tuesday (yes, flying all day xmas). But even so, a driver at 13:30 would be way too late to get to the airport. I ask the receptionist to point him out or describe him. She's flustered, trying to explain that he's black. (Black people are incredibly rare here.) Went out, found him, and after a confusing conversation, turns out that not only was he going to the wrong airport (Domodedovo, not Sheremetyevo), but he was supposed to be driving Boy George, not me. Wtf? I can only imagine the confusion at the receptionist's desk that she called me down to "my driver" instead of Boy George.
Anyway, Moscow is great, as always. I love this city. One warning, though: when a relatively large Russian guy with the nickname "Wolf" wants to drink whiskey with you, politely decline.
Speaking of wolves, apparently a wolf boy is loose in Moscow. Wtf.
Newly discovered funny blog from article above: Moscow Doesn't Believe in Tears.
Unrelated misc links:
- 2007 OpenID wrapup
- Who mattered in 2007 -- apparently I matter, and I'm an introvert. Several of my friends matter too. Love the insular valley. Reminds me of the directed graph I saw recently with two nodes labeled "Blog" with two edges pointing at each other. Yay! :)
|
|
|
| navigation |
| [ |
viewing |
| |
most recent entries |
] |
| |
|
|
