Sunday, January 16, 2011

How Facebook Ships Code

I'm fascinated by the way Facebook operates.  It's a very unique environment, not easily replicated (nor would their system work for all companies, even if they tried).  These are notes gathered from talking with many friends at Facebook about how the company develops and releases software.

Seems like others are also interested in Facebook...   The company's developer-driven culture is coming under greater public scrutiny and other companies are grappling with if/how to implement developer-driven culture.   The company is pretty secretive about its internal processes, though.  Facebook's Engineering team releases public Notes on new features and some internal systems, but these are mostly "what" kinds of articles, not "how"...  So it's not easy for outsiders to see how Facebook is able to innovate and optimize their service so much more effectively than other companies.  In my own attempt as an outsider to understand more about how Facebook operates, I assembled these observations over a period of months.  Out of respect for the privacy of my sources, I've removed all names and mention of specific features/products.  And I've also waited for over six months to publish these notes, so they're surely a bit out-of-date.   I hope that releasing these notes will help shed some light on how Facebook has managed to push decision-making "down" in its organization without descending into chaos...  It's hard to argue with Facebook's results or the coherence of Facebook's product offerings.  I think and hope that many consumer internet companies can learn from Facebook's example.

HUGE thanks to the many folks who helped put together this view inside of Facebook.   Thanks are also due to folks like epriest and fryfrog who have written up corrections and edits.

Notes:

  • as of June 2010, the company has nearly 2000 employees, up from roughly 1100 employees 10 months ago.  Nearly doubling staff in under a year!

  • the two largest teams are Engineering and Ops, with roughly 400-500 team members each.  Between the two they make up about 50% of the company.

  • product manager to engineer ratio is roughly 1-to-7 or 1-to-10

  • all engineers go through 4 to 6 week "Boot Camp" training where they learn the Facebook system by fixing bugs and listening to lectures given by more senior/tenured engineers.  estimate 10% of each boot camp's trainee class don't make it and are counseled out of the organization.

  • after boot camp, all engineers get access to live DB (comes with standard lecture about "with great power comes great responsibility" and a clear list of "fire-able offenses", e.g., sharing private user data)

  • [EDIT thx fryfrog] "There are also very good safe guards in place to prevent anyone at the company from doing the horrible sorts of things you can imagine people have the power to do being on the inside. If you have to "become" someone who is asking for support, this is logged along with a reason and closely reviewed. Straying here is not tolerated, period."



  • any engineer can modify any part of FB's code base and check-in at-will

  • very engineering driven culture.  "product managers are essentially useless here." is a quote from an engineer.  engineers can modify specs mid-process, re-order work projects, and inject new feature ideas anytime.  [EDITORIAL] The author of this blog post is a product manager, so this sentiment really caught my attention.  As you'll see in the rest of these notes, though, it's apparent that Facebook's culture has really embraced product management practices so it's not as though the role of product management is somehow ignored or omitted.  Rather, the culture of the company seems to be set so that *everyone* feels responsibility for the product.

  • during monthly cross-team meetings, the engineers are the ones who present progress reports.  product marketing and product management attend these meetings, but if they are particularly outspoken, there is actually feedback to the leadership that "product spoke too much at the last meeting."  they really want engineers to publicly own products and be the main point of contact for the things they built.

  • resourcing for projects is purely voluntary.

    • a PM lobbies group of engineers, tries to get them excited about their ideas.

    • Engineers decide which ones sound interesting to work on.

    • Engineer talks to their manager, says "I'd like to work on these 5 things this week."

    • Engineering Manager mostly leaves engineers' preferences alone, may sometimes ask that certain tasks get done first.

    • Engineers handle entire feature themselves -- front end javascript, backend database code, and everything in between.  If they want help from a Designer (there are a limited staff of dedicated designers available), they need to get a Designer interested enough in their project to take it on.  Same for Architect help.  But in general, expectation is that engineers will handle everything they need themselves.



  • arguments about whether or not a feature idea is worth doing or not generally get resolved by just spending a week implementing it and then testing it on a sample of users, e.g., 1% of Nevada users.

  • engineers generally want to work on infrastructure, scalability and "hard problems" -- that's where all the prestige is.  can be hard to get engineers excited about working on front-end projects and user interfaces.  this is the opposite of what you find in some consumer businesses where everyone wants to work on stuff that customers touch so you can point to a particular user experience and say "I built that."  At facebook, the back-end stuff like news feed algorithms, ad-targeting algorithms, memcache optimizations, etc. are the juicy projects that engineers want.



  • commits that affect certain high-priority features (e.g., news feed) get code reviewed before merge. News Feed is important enough that Zuckerberg reviews any changes to it, but that's an exceptional case.

  • [CORRECTION -- thx epriest] "There is mandatory code review for all changes (i.e., by one or more engineers). I think the article is just saying that Zuck doesn't look at every change personally."

  • [CORRECTION thx fryfrog] "All changes are reviewed by at least one person, and the system is easy for anyone else to look at and review your code even if you don't invite them to. It would take intentionally malicious behavior to get un-reviewed code in."

  • no QA at all, zero.  engineers responsible for testing, bug fixes, and post-launch maintenance of their own work.  there are some unit-testing and integration-testing frameworks available, but only sporadically used.

  • [CORRECTION thx fryfrog] "I would also add that we do have QA, just not an official QA group. Every employee at an office or connected via VPN is using a version of the site that includes all the changes that are next in line to go out. This version is updated frequently and is usually 1-12 hours ahead of what the world sees. All employees are strongly encouraged to report any bugs they see and these are very quickly actioned upon."

  • re: surprise at lack of QA or automated unit tests -- "most engineers are capable of writing bug-free code.  it's just that they don't have an incentive to do so at most companies.  when there's a QA department, it's easy to just throw it over to them to find the errors."  [EDIT: please note that this was subjective opinion, I chose to include it in this post because of the stark contrast that this draws with standard development practice at other companies]

  • [CORRECTION thx epriest] "We have automated testing, including "push-blocking" tests which must pass before the release goes out. We absolutely do not believe "most engineers are capable of writing bug-free code", much less that this is a reasonable notion to base a business upon."



  • re: surprise at lack of PM influence/control -- product managers have a lot of independence and freedom.  The key to being influential is to have really good relationships with engineering managers.  Need to be technical enough not to suggest stupid ideas.  Aside from that, there's no need to ask for any permission or pass any reviews when establishing roadmaps/backlogs.  There are relatively few PMs, but they all feel like they have responsibility for a really important and personally-interesting area of the company.

  • by default all code commits get packaged into weekly releases (tuesdays)

  • with extra effort, changes can go out same day

  • tuesday code releases require all engineers who committed code in that week's release candidate to be on-site

  • engineers must be present in a specific IRC channel for "roll call" before the release begins or else suffer a public "shaming"

  • ops team runs code releases by gradually rolling code out

    • facebook has around 60,000 servers

    • there are 9 concentric levels for rolling out new code

    • [CORRECTION thx epriest] "The nine push phases are not concentric. There are three concentric phases (p1 = internal release, p2 = small external release, p3 = full external release). The other six phases are auxiliary tiers like our internal tools, video upload hosts, etc."

    • the smallest level is only 6 servers

    • e.g., new tuesday release is rolled out to 6 servers (level 1), ops team then observes those 6 servers and make sure that they are behaving correctly before rolling forward to the next level.

    • if a release is causing any issues (e.g., throwing errors, etc.) then push is halted.  the engineer who committed the offending changeset is paged to fix the problem.  and then the release starts over again at level 1.

    • so a release may go thru levels repeatedly:  1-2-3-fix. back to 1. 1-2-3-4-5-fix.  back to 1.  1-2-3-4-5-6-7-8-9.



  • ops team is really well-trained, well-respected, and very business-aware.  their server metrics go beyond the usual error logs, load & memory utilization stats -- also include user behavior.  E.g., if a new release changes the percentage of users who engage with Facebook features, the ops team will see that in their metrics and may stop a release for that reason so they can investigate.

  • during the release process, ops team uses an IRC-based paging system that can ping individual engineers via Facebook, email, IRC, IM, and SMS if needed to get their attention.  not responding to ops team results in public shaming.

  • once code has rolled out to level 9 and is stable, then done with weekly push.

  • if a feature doesn't get coded in time for a particular weekly push, it's not that big a deal (unless there are hard external dependencies) -- features will just generally get shipped whenever they're completed.

  • getting svn-blamed, publicly shamed, or slipping projects too often will result in an engineer getting fired.  "it's a very high performance culture".  people that aren't productive or aren't super talented really stick out.  Managers will literally take poor performers aside within 6 months of hiring and say "this just isn't working out, you're not a good culture fit".  this actually applies at every level of the company, even C-level and VP-level hires have been quickly dismissed if they aren't super productive.

  • [CORRECTION, thx epriest]  "People do not get called out for introducing bugs. They only get called out if they ask for changes to go out with the release but aren't around to support them in case something goes wrong (and haven't found someone to cover for you)."

  • [CORRECTION, thx epriest] "Getting blamed will NOT get you fired. We are extremely forgiving in this respect, and most of the senior engineers have pushed at least one horrible thing, myself included. As far as I know, no one has ever been fired for making mistakes of this nature."

  • [CORRECTION, thx fryfrog] "I also don't know of anyone who has been fired for making mistakes like are mentioned in the article. I know of people who have inadvertently taken down the site. They work hard to fix what ever caused the problem and everyone learns from it. The public shaming is far more effective than fear of being fired, in my opinion."


It'll be super interesting to see how Facebook's development culture evolves over time -- and especially to see if the culture can continue scaling as the company grows into the thousands-of-employees.

What do you think?  Would "developer-driven culture" work at your company?

287 comments:

  1. Wonder what exactly Product Managers do....

    ReplyDelete
  2. The cubicle is the result of culture driven work environment. Intel used it so that engineers could have privacy yet communicate if they needed to rapidly. Its interesting that other companies simply employ cubicles because of the successful nature of intel, not necessarily because of what it actually does to improve the work environment or improve the operations of the company.

    ReplyDelete
  3. There are some significant inaccuracies in this article. See my post on reddit for some clarifications:

    http://www.reddit.com/r/programming/comments/f3u0n/how_facebook_ships_code/c1d3b37

    (I do not speak for my employer, Facebook.)

    ReplyDelete
  4. Thanks for the clarifications re: mandatory code review, automated tests, and emphasis on bug fixing. Sounds like Facebook's practices are actually more defensive around code checkins than my notes may indicate.

    The thing I still find remarkable is how all of those practices (and it seems like the rest of the culture) centers on engineers...

    ReplyDelete
  5. [...] Apparently, Facebook has around 60,000 servers. How are there ever times when Facebook is down? [...]

    ReplyDelete
  6. So what's the other half of the company, and what all do they do? How do these other units get on with Engineering and with Ops?

    ReplyDelete
  7. You know, designers do actually have a purpose. Especially on a site like Facebook.

    I hate to say this, because Facebook's approach seems to generally work pretty well. Most engineers (read: developers) have little to no grasp of the user experience. And in terms of Facebook causing widespread user experience panic every time they shuffle the layout, I'm not surprised to learn that FB allows engineers to design the user experience most of the time. No wonder the layout this time around especially sucks...engineers are designing in a linear fashion.

    ReplyDelete
  8. [...] I wonder if this method really does scale well. I don’t think it would work everywhere. [...]

    ReplyDelete
  9. The multi-level release cycle is interesting, but how does Facebook deal with database schema changes etc? Are they simply using a really great key->value nosql type of solution?

    ReplyDelete
  10. I most areas responsability is shifting towards engineers generally with the introduction of agile methodologies, and the more successful companies who are able to hire more talented, well rounded engineers are generally able to carry this off more effectively.

    I think such practices become harder to maintain as the work force grows.

    ReplyDelete
  11. If all the engineers write bug free code or at least caught in the release process where they are paged to fix their issues or face public shaming or at worst getting fired then why are boot campers only dedicated to fixing bugs and listening to lectures? Wouldn't that mean the boot campers have nothing to do but listen to lectures?

    ReplyDelete
  12. Yee, you should really correct this post based on Evan's corrections.

    ReplyDelete
  13. Interesting notes given what a hair-pulling nightmare their public APIs are to integrate with. Stuff changes regularly and gets dropped on apparent whims with no prior warning so it makes perfect sense that the development is run from the bottom up with no overarching vision to ensure that not just the sexy features du jour are getting attention.

    ReplyDelete
  14. [...] this post today on Frame Think: http://framethink.wordpress.com/2011/01/17/how-facebook-ships-code/ about how Facebook ships code. My favorite part: arguments about whether or not a feature idea is [...]

    ReplyDelete
  15. First it's either unique or not. It can't be "very unique"

    Secondly this process would seem to explain why Facebook seems to break its own APIs so often.

    ReplyDelete
  16. I would rather work at Google :)

    ReplyDelete
  17. [...] I was just reading this article found on Hacker News about the development process at facebook. [...]

    ReplyDelete
  18. [...] Story: How Facebook Ships Code) This entry was posted in web2.0 and tagged development, facebook. Bookmark the permalink. [...]

    ReplyDelete
  19. bullshit

    from line 1 to line {end}

    ReplyDelete
  20. I +1 a lot of that...

    no QA, check-in at will, everyone has access to production

    That is the peace through anarchy model.

    Rizzo

    ReplyDelete
  21. This pattern is repeated all the time for Process.

    A given rule/process should be implemented to solve a particular problem. If you don't have that problem, you don't need that process/tool.

    People need to be more pragmatic and impose the smallest number of rules/processes required.

    ReplyDelete
  22. Even with the corrections (re: the FB employee) I find the cultural implications at Facebook very interesting. I work in a company the requires intense engineering development, but still utilizes the cubicle model. I find that this is a major drain on productivity and slows our release schedule for a variety reasons (less camaraderie, less face time, lowered team cohesion, etc) . I'm sure that the FB culture isn't right for everyone, but I'm sure there is a nice midpoint between cubiclism and full-open culture.

    ReplyDelete
  23. [...] Facebook Ships Code http://bit.ly/enT9nP “with great power comes great responsibility”Five Emotions Invented By The Internet [...]

    ReplyDelete
  24. WPStats (wordpress stats) tracking bug image.

    ReplyDelete
  25. [...] Questions, Topics and PeopleAdd QuestionAdd QuestionAre product managers useless at Facebook?http://framethink.wordpress.com/...It does look so  BIU     @   Edit Link Text Show [...]

    ReplyDelete
  26. Are product managers useless at Facebook?...

    http://framethink.wordpress.com/2011/01/17/how-facebook-ships-code/ It does look so...

    ReplyDelete
  27. Agreed. The UX on facebook really sucks.

    ReplyDelete
  28. Perhaps they act as cheerleaders/soundingboards for ideas.

    ReplyDelete
  29. You know that, most of the layout changes are to do with a backend process, or to do with improving the performance metrics of teh system as a whole; holistic application development cannot be undertaken by designers as they know what they are asking for ;)

    ReplyDelete
  30. The layout is just fine. It's noobs like you that made MySpace popular in the first place.

    ReplyDelete
  31. How about you qualify that statement with some specific examples? I personally have no (significant) problem with FB's UI. It's rather straight-forward to me.

    I used to have trouble with privacy options but I figured that was intentional :)

    ReplyDelete
  32. A few ways:

    1) Organizing server is down, so those 60,000 servers all can't get to something.
    2) You only connect to one of those, and that might be down before their system can detect it.

    ReplyDelete
  33. re: Uniqueness - That depends on the scope of reference. An object can be considered "quite unique" if many of it's attributes are unique, but some are not.

    The syntax of human language evolves over time. Stop being pedantic else be left behind :)

    ReplyDelete
  34. "foobar Says:" Haha! That's absolutely right!

    ReplyDelete
  35. Have a look at this video, it has a great overview of facebook's architecture.
    http://www.techiegyan.com/2009/04/13/facebook%E2%80%99s-architecture/

    ReplyDelete
  36. [...] Even with the corrections (re: the FB employee) I find the cultural implications at Facebook very interesting. I work in a company the requires intense engineering development, but still utilizes the cubicle model. I find that this is a major drain on productivity and slows our release schedule for a variety reasons (less camaraderie, less face time, lowered team cohesion, etc) . I’m sure that the FB culture isn’t right for everyone, but I’m sure there is a nice midpoint between cubiclism and full-open culture. ~ Jack Amplify’d from framethink.wordpress.com [...]

    ReplyDelete
  37. [...] thorough look at how Facebook ships code. From internal engineering processes to the lack of project manager influence/control, how and [...]

    ReplyDelete
  38. [...] today, FrameThink posted an entry about how Facebook ships code. There were quite a lot of “juicy” bits to the story [...]

    ReplyDelete
  39. How accurate is the "How Facebook Ships Code" article written by 'yeeguy'?...

    Thanks a ton for the corrections and additional material. I think it'd be fantastic for more companies to learn about how Facebook's development process works -- "developer-driven culture" is really the ultimate expression of pushing decisions "do...

    ReplyDelete
  40. [...] January 17, 2011 by Brian McKay Leave a Comment I'm fascinated by the way Facebook operates.  It's a very unique environment, not easily replicated (nor would their system work for all companies, even if they tried).  These are notes gathered from talking with many friends at Facebook about how the company develops and releases software. It's been over six months since I assembled these observations and I'm sure Facebook has continuously evolved its software development practices in the meanti … Read More [...]

    ReplyDelete
  41. [...] recomiendo la lectura del artículo completo: http://framethink.wordpress.com/2011/01/17/how-facebook-ships-code/ [...]

    ReplyDelete
  42. I think techniques like these may be very successful only in organizations where developers are working on consumer end-products that they themselves use (like Facebook). In that environment, the fundamental role that a product manager plays of identifying, prioritizing and translating the needs of end-users is more easily obviated. I can't see this working on software projects where the problem domain is much more complex or esoteric.

    ReplyDelete
  43. Great insight, thank you. I am surprised most of all about the lack of unit testing and formal QA department. I'd have to work in such an environment to give a fair opinion, but it sounds like the wild west for quality. Their stability would seem to prove me wrong though.

    ReplyDelete
  44. Well-well look. I already told you: I deal with the god damn customers so the engineers don't have to. I have people skills; I am good at dealing with people. Can't you understand that? What the hell is wrong with you people?

    ReplyDelete
  45. that's pretty ignorant. a good product manager is often the font of good ideas, not just the first defense against bad ones.

    ReplyDelete
  46. This is fascinating. I would love to work for a company that has that kind of culture.

    ReplyDelete
  47. The semantics of human language changes far less over a short period of time.

    And for the word unique, it hasn't changed at all in many years.

    Stop being an idiot and perhaps you'll have a fighting chance to catch up to those who are actually _in_ the race.

    ReplyDelete
  48. [...] How Facebook Ships Code I’m fascinated by the way Facebook operates.  It’s a very unique environment, not easily replicated (nor would their [...] [...]

    ReplyDelete
  49. [...] How Facebook Ships Code « FrameThink – Frameworks for Thinking People interesting [...]

    ReplyDelete
  50. [...] ran across this article today on FrameThink about Facebook’s deployment process. For a company that has grown and [...]

    ReplyDelete
  51. whoa. look at the amazing work environment. all that focus on product and how powerful engineers are. it really shows in how profitable facebook is. their balance sheet is a glowing example of the success of this company.

    ReplyDelete
  52. Good point John. The other item that's overlooked is the fact that product managers are less important when you have lots of money to burn on "experiments" and the BEST engineers who are capable of understanding what users want and what features to build/prioritize. Facebook has that luxury today, as did Google a couple years ago. When the quality of engineering talent goes down, the importance of a talented product team goes up. Historically, you can see that shift at companies like Yahoo, and you can see it starting to happen at Google now (guys like Max Levchin and Brad Horowitz are undoubtably 'product' leaders who are hired when engineers can't experiment their way through things). Lastly, it's undeniable that Zuck is a 'product' guy just as he is an 'engineer' - this is common with most of the major web 2.0 companies.

    ReplyDelete
  53. Ok, so... I'm a noob because I believe that engineers make terrible front-end designers? Engineers think logically and end-users generally don't. So to have a group of people (engineers) changing the layout to fit back-end needs, rather than the needs of the people who'll actually be using it is absurd. Especially when you're talking about 400 million users.

    ReplyDelete
  54. I hope you're joking, Opdelt.

    ReplyDelete
  55. [...] I'm fascinated by the way Facebook operates.  It's a very unique environment, not easily replicated (nor would their system work for all companies, even if they tried).  These are notes gathered from talking with many friends at Facebook about how the company develops and releases software. It's been over six months since I assembled these observations and I'm sure Facebook has continuously evolved its software development practices in the meanti … Read More [...]

    ReplyDelete
  56. It's pretty rare to find an engineer that has a decent grasp of the complexities and subtleties of Ux and UI design but only the best engineers will not kid themselves that they do and actually respect people that do (equally so inversely).

    FB's Ux/UI is flawed on many levels, breaks a bunch of industry standards and rules of thumb but that does not render it useless by any means, it just will prevent a percentage of users from accessing certain features/functionality easily. If these are non-critical areas then it is less of an issue than say "privacy settings" for example.

    ReplyDelete
  57. Haha! Excellent timing sir.

    ReplyDelete
  58. [...] stuff here.   If you enjoyed this article, please consider sharing [...]

    ReplyDelete
  59. [...] How Facebook Ships Code – “I’m fascinated by the way Facebook operates. It’s a very unique environment, not easily replicated (nor would their system work for all companies, even if they tried). These are notes gathered from talking with many friends at Facebook about how the company develops and releases software….” [...]

    ReplyDelete
  60. If its an Engineering driven environment, what role do Product Managers play? Seriously.

    ReplyDelete
  61. [...] back-end of the social network Facebook.com is complex. Facebook’s servers run on a LAMP stack with Memcache. They also use MySQL databases and PHP, [...]

    ReplyDelete
  62. Nice quote from Office Space.

    Didn't that guy get hit by a bus later on?

    ReplyDelete
  63. facebook sucks. myspace sucked. people only use it because their friends use it. everyone left myspace and everyone can leave FB just as easily.

    no designers and no real QA?

    wtf?

    ReplyDelete
  64. There is no modifier for unique. It's just unique. Not "very unique", not "really unique".

    ReplyDelete
  65. [...] they should know a bit about the science of pricing by nowHow Facebook Ships Code http://bit.ly/enT9nP “with great power comes great responsibility”Five Emotions Invented By The Internet [...]

    ReplyDelete
  66. [...] Read the rest of this post on the original site Tagged: Facebook, digital, frontpage, social networking, software, FrameThink, organization, Yee Lee | permalink var SurphaceSettings = { url: "http://voices.allthingsd.com/20110118/how-facebook-ships-code/", siteid: "atd" }; var _surphld = document.createElement("script"); _surphld.type = "text/javascript"; _surphld.src = "http://cdn11.surphace.com/rcwidget/loader.js"; (document.getElementsByTagName("head")[0] || document.getElementsByTagName("body")[0]).appendChild(_surphld); « Previous Post Next Post » ord=Math.random()*10000000000000000; document.write(''); [...]

    ReplyDelete
  67. I guess FB engineers should be not too happy about the article as it might look they are a bunch of crazy wild-west coders. Do not believe they do not have a discipline or that PMs or MKT do not play a role, might it be different from other companies standards.
    Engineers involvement can be good or bad. I see some positive aspects like being responsible for their code and features, fact that empowers them, but things like that they decide the products features just give me the frights.
    As a final comment, I think you are looking for a silver bullet that reading SW-Eng savvy people like Tom deMarco in "Peopleware" you can conclude does not exist. SWEng practices are there for a reason and a SW process is always needed, be it a formal RUP process or an Agile one. BTW, Agile does not mean "do what I want". It is a formal process like any other, with its disciplines and musts.
    Problem is many people out there just cover their anarchy and lack of process stating " I am agile".
    Honestly do not think the process described is magic or a novelty, although has attractive bits. Hope it works fine for them.

    ReplyDelete
  68. This article is obviously written by a developer frustrated with pm's, with sources of developers who get a huge hard on by the power and responsiblity they THINK they have. This is just not a realistic scenario.

    ReplyDelete
  69. [...] purpose or not, there is no denying that Facebook is a high traffic, technically challenging site. These rumors on how Facebook handles development and deployment are of course to take with a grain of salt, but [...]

    ReplyDelete
  70. If the UX is so bad, I wonder why so many sites try to copy Facebook's, and why I know so many UXler who love FB.

    And: There are designers (and all should be able to do UX also, I do not see why these fields should be separated), if the developers think they need any. In most cases a general template is used however, and the developers can copy the structure and do the rest rather easily.

    ReplyDelete
  71. [...] Chan at 18:35 (7 seconds ago) in Blog No Response9 viewsInteresting blog post by yeeguy entitled How Facebook Ships Code. mig33 do share some common things with Facebook =)Here are 8 interesting snippets:All engineers go [...]

    ReplyDelete
  72. From my experience, this model always works, BUT:
    -you need a team of excellent engineers (all of them, no exceptions)
    -you need to convince management that it is the way to go.

    So, in the real world, it is seldom used because:
    -great engineers cost more money (and they just write code right? they're all the same)
    -management rarely has an engineering background to understand why this model is worth it

    ReplyDelete
  73. I'd hate to work at Facebook based on the whole shaming thing, they'll kill all their engineers with stress.

    ReplyDelete
  74. I've worked in an organisation that had a similar setup.

    It was a disaster so much so that the organisation is now refocusing very strongly on a product management culture.

    To me reading this suggests Facebook have too many developers and too little focus on quality and doing right by their users. No surprise that they frequently have users up in arms about UI and privacy.

    You need balance in an organisation to get the best out of your team. Putting all the emphasis on developers does not give a balance of skills.

    ReplyDelete
  75. You forgot to mention how they ship their code technically:
    The use BitTorrent! Every updated server will be another seeder. it´s pure genius if you have thousand of servers to update.

    ReplyDelete
  76. [...] How Facebook Ships Code « FrameThink – Frameworks for Thinking People (tags: to_read facebook development programming management code) [...]

    ReplyDelete
  77. [...] 原文在此,看完之后,终于明白为什么优秀的工程师都去了/想去facebook,因为那里是工程师们的天堂。 [...]

    ReplyDelete
  78. [...] recently read a great article about how Facebook ships code and what their values and practices are. It revolves around a list of different examples of how [...]

    ReplyDelete
  79. Sounds hideous to my mind. Similarly working for ferrari must be boring. Striving for perfection in the mundane. I prefer concept level and cant quite get excited about this but judging by all the comments I may be alone :) and happy about it.

    ReplyDelete
  80. [...] read two interesting articles about development in Facebook (How Facebook Ships Code and LAUNCH002: What I Learned from Zuckerberg’s Mistakes). Apparently the programmers choose [...]

    ReplyDelete
  81. [...] encontrado un interesante post en el blog FrameThink, en el cuál su autor que usa un pseudónimo "yeeguy", dice que ha tenido [...]

    ReplyDelete
  82. I wonder how User Experience/Usability people work within an organization like that. Typically, the fact that the engineers have the final words would probably turn off many UX practitioners who may feel that the work/studies they are doing and the findings they are proposing does not get listen to.

    ReplyDelete
  83. Hambone, are you that culturally illiterate that you did not pick up on the Office Space reference? Crawl back under your bowl of rice and or noodles and continue banging out code and let the "grown folks" continue this conversation.

    ReplyDelete
  84. I agree with you on that one. (see comment I posted today below). I mean, I have met a few developers who can do UX design, but the reality is, most of them can't. They just don't think in a user-centered manner, they think from the structure of the code. So basically, if you want a UI that shows you the structure of the code (or the organization of your data structures), developers are good to produce that. The problem is that users don't think that way. Thinking from a user-centered point of view is something different.

    Now, as for designers, yes I agree with the one of the post that says designers should be able to do the UX (after all they are designers). The problem here again is that many of them never heard of UX in their design training. While they should not be UX specialist, they should have at least a minimum training in UX.

    At the end of the day, it all comes down to how capable is someone (in any position) to think from a user-centered point of view. History shows us that certain people are capable of doing this naturally (these are the very good designers, engineers, developers, trade workers, etc... but are rare). Most often though, people don't naturally think that way. I always use the example of an electrician when I teach my UI/UX class (nothing against BTW). If you hire an electrician to wire your house, you really want to hire the one who will think about how people in the house are going to move around and where switches, receptacles, and three-way switches should be located. That will lead to well-designed electrical system configurations that are usable by the people live in the house (the primary users). Do these people exist? Sure they do, in a very limited quantity. The experience most of us have when we move into our houses is that some switches are behind doors, some three-way switches are missing to allow you to turn off the lights at the end of the hallway, and a few receptacles are in usable spots. Yet, from a wiring point of view, everything is working... That does not mean electrician are not good at what they do. It just shows that their natural tendencies may not be to wire a house from a user-centered point of view.

    ReplyDelete
  85. I think UX need to get people excited too. So what if engineering own the front end. UX would need to get the engineers excited about making features more usable. So the engineer that is coding a new interaction could use a UX mindset.

    UX is a state of mind anyway, the UX team at Facebook could probably be a good catalyst role.

    ReplyDelete
  86. [...] especially not when you’re working in a team. So does Facebook handle this problem? Check out this interesting blogpost! [...]

    ReplyDelete
  87. It's easy to say that people love this and people love that, it's another thing to demonstrate it with a solid used-centered process. We all have opinions about what we love and what we hate, but "Your opinion, while interesting is often irrelevant". What's relevant is what UX practitioners can demonstrate through the data collected from real users of the product. I've been in this battle many times and it's what I consider to be the biggest problem with UI/UX design:

    Since everyone uses a computer, everyone thinks they are able to design a UI and have an opinion on the UI. That's the core of the problem here. If a civil engineer comes to you and makes a recommendation about a structure, most likely people will not debate, on the basis of his/her expertise (civil engineers may debate between them, but Joe and Jane from let's say social studies won't). When it comes to Human Factors Engineering/UX design, while there are specialists out there with the appropriate training who can make recommendations based on an established process, we all think we can have our say on the basis that we all used a UI at some point in our life. That's a very dangerous approach to follow.

    So why so many sites try to copy Facebook? Because they don't know better. They think if millions of people use it, it most be good, so let's do it. WRONG! It does not mean millions of people use it that it's a good design. Millions of people may have been forced to adapt to a bad design (humans have great abilities to adapt over time, even the worst design may not be too bad for some users because they have adapted to it).

    If you are carefully reading people's comments on facebook about their UI when major UX overhauls are pushed through, you'll read a lot of things like "OH NO, FACEBOOK UI CHANGES AGAIN #$%*&&#$" and after a few days, people painfully adapt to the new UI. Not the best approach to push a new UI through if you ask me. That being said, not every aspect of their UI is like that. There are definitely good aspects.

    ReplyDelete
  88. [...] Want to know a little about Facebook behind the scenes? [...]

    ReplyDelete
  89. This will not work on all organizations, however, some of the ideas presented here are already well practiced in a number of places - I for example have been doing most of these for the last 12 years at different organizations. I got to a point where I did 63 releases in one year (medium and mayor, small where par-for-the-course)

    I like that the systems guys are business oriented and well respected. In most organizations that is not the case. I have always struggled with that and solved it in couple of places but not in all places I have worked for.

    I am not sure about the lack of QA. Maybe I am old school about that. I do agree that developers are capable of writing solid code and they should totally and completely unit test the heck out of the code they write - take ownership of what you do for goodness sakes - but I still feel that a formal QA step is of great importance.

    ReplyDelete
  90. [...] read for the geeks and developers out there: How Facebook ships and develops its code. I like the “Engineers Rule” culture of the [...]

    ReplyDelete
  91. Companies use cubicles because they count as furniture come tax-time instead of counting as construction of offices.

    ReplyDelete
  92. [...] Camp” training where they learn the Facebook system by fixing bugs. After boot camp, all engineers get access to live DB and any engineer can modify any part of Facebook’s code base and check-in at-will so that [...]

    ReplyDelete
  93. haha... development driven culture doesn't exist in any tech company in the financial industry... i can only dream of the day when i can check in at will... perhaps when i'm outta this place.

    ReplyDelete
  94. [...] How Facebook Ships Code (tags: facebook development programming culture management) [...]

    ReplyDelete
  95. [...] Camp” training where they learn the Facebook system by fixing bugs. After boot camp, all engineers get access to live DB and any engineer can modify any part of Facebook’s code base and check-in at-will so that [...]

    ReplyDelete
  96. [...] I'm fascinated by the way Facebook operates.  It's a very unique environment, not easily replicated (nor would their system work for all companies, even if they tried).  These are notes gathered from talking with many friends at Facebook about how the company develops and releases software. Seems like others are also interested in Facebook…   The company's developer-driven culture is coming under greater public scrutiny and other companies are … Read More [...]

    ReplyDelete
  97. Wait, we're arguing about a site who's user primary interface is a single text field right?

    ReplyDelete
  98. [...] Shared How Facebook Ships Code « FrameThink – Frameworks for Thinking People. [...]

    ReplyDelete
  99. Aha, this is what I call a company 100% SOX compliance!

    I wouldn't put my money here, at all.

    ReplyDelete
  100. [...] Camp” training where they learn the Facebook system by fixing bugs.After boot camp, all engineers get access to live DB and any engineer can modify any part of Facebook’s code base and check-in at-will so that [...]

    ReplyDelete
  101. [...] the same time, a blog post on “How Facebook ships code” has been making the rounds, claiming to detail the company’s “hacker” culture (hacker [...]

    ReplyDelete
  102. sounds like organised mayhem

    ReplyDelete
  103. @nithin sounds like you haven't experienced any good product managers. they are often the only people in the organisation that can simultaneously understand the complexities of the technical effort, the needs of the market, and the nuances of the user experience, whilst providing balance to some of the egos that can quickly torpedo a viable business.

    ReplyDelete
  104. [...] Manager Yee Lee talked to a bunch of friends who work at Facebook, and his resulting post on how Facebook ships code makes it sound like the biggest startup in the world. Some of the points in the original post were [...]

    ReplyDelete
  105. [...] Manager Yee Lee talked to a bunch of friends who work at Facebook, and his resulting post on how Facebook ships code makes it sound like the biggest startup in the world. Some of the points in the original post were [...]

    ReplyDelete
  106. I'm curious to understand the function of architecture at FB. To level set, I run enterprise Architecture at my company and I'm constantly seeking clarity on how architecture and architects enable the business and engineering. The article briefly alludes to architecture in FB as an advisory function akin to the designer function being a support function when engineering needs them. So...what do architects get called upon by the engineers to help them out with.

    Traditionally, the word "architect" is viewed very negatively by engineers - something I'm keenly aware of. How is this at FB?

    ReplyDelete
  107. This is one of the best "programming in the real world" posts I have read. Great job!

    ReplyDelete
  108. [...] boot camp, all engineers get access to live DB and any engineer can modify any part of Facebook’s code base and check-in at-will so that [...]

    ReplyDelete
  109. [...] 原文在此,看完之后,终于明白为什么优秀的工程师都去了/想去facebook,因为那里是工程师们的天堂。 [...]

    ReplyDelete
  110. [...] boot camp, all engineers get access to live DB and any engineer can modify any part of Facebook’s code base and check-in at-will so that [...]

    ReplyDelete
  111. This will work when the following has been taken care of:
    - business strategy & missions clear
    - architects have defined the strategic technology direction to be taken, and identified key technologies that need to be used
    - the company is able to hire & retain highly competent developers over a period of time
    - focus & accountability areas are defined for business, architects, developers

    ReplyDelete
  112. The layout that FB have now is very very good for the users.

    ReplyDelete
  113. sounds like something that would work great until the big buyout/ipo, at which point people would stop caring, the threat of being fired wouldn't carry much weight and i suspect things would go to utter and complete hell.

    ReplyDelete
  114. [...] 英文原文:How Facebook Ships Code 中文翻译:Facebook 是如何管理代码的 [...]

    ReplyDelete
  115. [...] Facebook Ships Code coding, startups, facebook, culture, developer-driven culture... [full post] yeeguy FrameThink - Frameworks for Thinking People business managementproduct [...]

    ReplyDelete
  116. [...] I'm fascinated by the way Facebook operates.  It's a very unique environment, not easily replicated (nor would their system work for all companies, even if they tried).  These are notes gathered from talking with many friends at Facebook about how the company develops and releases software. Seems like others are also interested in Facebook…   The company's developer-driven culture is coming under greater public scrutiny and other companies are … Read More [...]

    ReplyDelete
  117. What does Facebook use to package and deploy PHP code? We currently use git for deployment at my company but we've come to the conclusion that using a packaged deployment system would be much better, especially for our cloud-based systems.

    ReplyDelete
  118. [...] 英文原文:How Facebook Ships Code 中文翻译:Facebook 是如何管理代码的 [...]

    ReplyDelete
  119. [...] I'm fascinated by the way Facebook operates.  It's a very unique environment, not easily replicated (nor would their system work for all companies, even if they tried).  These are notes gathered from talking with many friends at Facebook about how the company develops and releases software. Seems like others are also interested in Facebook…   The company's developer-driven culture is coming under greater public scrutiny and other companies are … Read More [...]

    ReplyDelete
  120. [...] 原文在此,看完之后,终于明白为什么优秀的工程师都去了/想去facebook,因为那里是工程师们的天堂。 [...]

    ReplyDelete
  121. billy_ripken, are you that egocentric, intellectual snob, that you call everybody who doesn't remember every quote from movies you like a "cultural illiterate"?

    ReplyDelete
  122. [...] How Facebook Ships Code Interessante Einblicke [...]

    ReplyDelete
  123. The FB engineers remind me of another group of R&D engineers in Kenya working on GWT UI for ADempiere Opensource ERP client http://www.at.co.ke/blog/trend-analysis/a1erp-the-web-2-0-erp%e2%80%93-the-technology-the-challenge
    Engineers can truly drive UI developments but PM's still remain key to guild the overall vision of the development.

    ReplyDelete
  124. So it all comes to this:
    Security is not important.
    Privacy of the user is not important.
    Is is fun to program.

    A developer driven development cycle is not the way to create a useful application. They teach this a long time ago already.
    Thank you for sharing this information. Facebook development is not the way to create applications.

    ReplyDelete
  125. If the Engineers don't think that their managers do much, and that the Engineers have a lot of freedom, and that the managers aren't very influential, then my guess is that the managers are really excellent managers! Their job is to keep their team as productive as possible. Believe me, I am sure that the managers are doing TONS of work behind the scenes to let the developers work in the best possible environment and to feel the best about what they do. Those managers are keeping things together!

    ReplyDelete
  126. Managers in general seem pretty worthless to me.

    ReplyDelete
  127. Offices can also be counted if you know what you are doing... This guy does.

    http://www.linkedin.com/pub/frank-little/12/912/513

    ReplyDelete
  128. Olivier,
    Rather than being so long winded, why not just give example sites with a good interface. I think Facebook has a wonderful interface, one of the best out there, and that is why I use it everyday.

    ReplyDelete
  129. I do not think so, In my opinion it would hardly work anywhere else. Why ? it all about business strategy, from what i understood, facebook need creativity, and that culture also instill a sense of leadership in each employee, which kind of ensures that only good ideas transform into code. Whereas in most businesses, programming must be aligned with business goals( which comes down from the top, to make sure everything is well coordinated) which somehow kills creativity. what do u guys think?

    ReplyDelete
  130. [...] dream to be a Facebook  developer and be recruited by Mark Zuckerberg? I advice you to take a look here (just for developers). LikeBe the first to like this [...]

    ReplyDelete
  131. [...] Lee在博客上发表了一篇名为How Facebook Ships Code的文章,迅速登上Reddit和Hacker [...]

    ReplyDelete
  132. I remember when I couldn't create an event in facebook because the state dropdown was empty and yet it was "required"...that remained for days and really sucked! As a developer I am amazed that facebook prospers as it does, it really doesn't seem so special - beyond having a massive user base. I don't even find it particularly intutive but I am glad they exist and give the web more press!

    ReplyDelete
  133. I agree with thepiedpipes. Project managers are very useful to a developer if they are good at what they do. If you need answers to something they go get it, if you don't want to talk to some annoying account exec just refer them to the PM. Personally they make great "meat shields" and "go getters". They have a tough job. They do everything in a project that no one else wants to do. Having a good PM, I can concentrate on the quality of my code and I can trust that everything else is taken care of. But, I've also had bad PMs and a bad PM just makes the project worse that if there weren't one at all.

    ReplyDelete
  134. [...] That’s one of the things that got me while reading How Facebook Ships Code « FrameThink – Frameworks for Thinking People [...]

    ReplyDelete
  135. facebook doesn't strike me as a terribly complex system except for it's scalability. I don't think this process would work if they were building an office suite or something like that.

    It's also the worst development environment I've ever seen or even heard of.

    I have an EXE file I built in turbo pascal in 1983 and that same binary will still run on the lastest version of windows on the latest hardware.

    I wrote a facebook app and within 3 months it wasn't working anymore because facebook doesn't understand the concept of backward compatibility. This doesn't strike me as a good feature of their culture being so entitled that they can break their api willy nilly without any regard for the work they're creating for their VOLUNTEER application developers.

    When the fad wears out they're going to find that people aren't so willing to cater to their every api change just because "It's facebook."

    ReplyDelete
  136. http://www.thinkgeek.com/books/humor/8e6c/images/2070/

    ReplyDelete
  137. [...] came an interesting article about Facebook’s internal development operations. It’s a great read. [...]

    ReplyDelete
  138. Disagree . . . imo something can be "very unique", if it's not just different from everything else, but *very* different from everything else.

    Agree though that "very unique" is heavily abused, and in most cases is redundant.

    ReplyDelete
  139. I hope this article isn't accurate because it makes it sound like a real cackhanded way to run things. I mean I get that it sounds like Mark has made his website and is idealistic about the way he wants to keep the power in the devs hands but come on!

    People have specialities, there is toooooo much to learn for a single dev. Making everyone do everything from db through to UI? I'm a developer, but as a regular user of Facebook I'm kind of disappointed at this haphazard way of doing new features. I mean apart from the fact that there aren't that many new big new features, but I kind of expected there to be some real deep knowledge going into this. Like psychology professors thinking about person to person interactions. And dedicated people that take care of the scaling problems that arise when you are developing for 500 million users.

    I cant believe this is really the way that Facebook is run, there must be some broader picture that isn't being shown here. Are you seriously saying that Facebook dev division is 500 people just doing what they want? I really want to believe that the best minds in their fields are coming together to produce the website that shapes the lives of so many people.

    Even if they have really high quality developers on the team they just simply cannot gain the depth of knowledge to offer the best if they do the whole gamut. We've all heard Seth's theories of 10,000 hours before you are an expert. Either all the developers are over 60 or they just simply are not experts in what they do.

    I can only guess that the real picture of this is that anyone can seed a feature idea but then teams take over various sub aspects of that feature and apply their various expertise to say, fleshing out the UI design, or optimising the caching, until really its just a nice notion that some developer "invented" a feature, just the same as I'm sure 99% of the code that Mark originally wrote has been replaced by now.

    ReplyDelete
  140. [...] they should know a bit about the science of pricing by nowHow Facebook Ships Code http://bit.ly/enT9nP “with great power comes great responsibility” Powered by Fresh From This entry was posted in [...]

    ReplyDelete
  141. Sounds like the screenplay for "The Social Network 2.0"

    ReplyDelete
  142. [...] 原文在此,看完之后,终于明白为什么优秀的工程师都去了/想去facebook,因为那里是工程师们的天堂。 [...]

    ReplyDelete
  143. [...] that exposed user data, and the user IDs that rogue apps associated with personal information, every damn engineer at Facebook has access to the database? That’s right, all one thousand engineers can look at [...]

    ReplyDelete
  144. I don't know if you heard, but most of facebook is written in php. Not exactly the language of geniuses. From what I've read, there's a group of heavy hitters that wrote the php compiler and the javascript reinterpreter thing, but they pride themselves on being wide and short rather than thin and tall if you get what I mean. These guys aren't writing compilers, or 3d rendering engines, they're moving text from the database to the screen.

    ReplyDelete
  145. [...] 原文在此,看完之后,终于明白为什么优秀的工程师都去了/想去facebook,因为那里是工程师们的天堂。 [...]

    ReplyDelete
  146. [...] I'm fascinated by the way Facebook operates.  It's a very unique environment, not easily replicated (nor would their system work for all companies, even if they tried).  These are notes gathered from talking with many friends at Facebook about how the company develops and releases software. Seems like others are also interested in Facebook…   The company's developer-driven culture is coming under greater public scrutiny and other companies are … Read More [...]

    ReplyDelete
  147. [...] facebook是如何管理代码的 Posted by Niko on January 20, 2011 in Reprinted. 0 Comments Tags: facebook. var jiathis_config = {"data_track_clickback":true}; 原文在此,看完之后,终于明白为什么优秀的工程师都去了/想去facebook,因为那里是工程师们的天堂。 [...]

    ReplyDelete
  148. [...] 原文: http://framethink.wordpress.com/2011/01/17/how-facebook-ships-code/ [...]

    ReplyDelete
  149. [...] 英文原文:How Facebook Ships Code This entry was posted in 流言蜚语. Bookmark the permalink. ← 思考 [...]

    ReplyDelete
  150. RE Dave G: I thought the point here was to discuss how does UX fits in the facebook development process described in the article rather than comment on the facebook UI

    ReplyDelete
  151. [...] 英文原文:How Facebook Ships Code 中文翻译:Facebook 是如何管理代码的 [...]

    ReplyDelete
  152. The inmates are running the asylum!
    This goes a long ways to explaining the missteps and lack of real user experience that Facebook has.
    I have over 31 years of experience building commercial software products and I can tell you what’s described here is a recipe for disaster in the long run.
    This sort of cowboy culture can work early in the development conceptual stage of product development but it’s no way to run a business.
    Developers are not your typical users and are generally terrible at building usable products.

    ReplyDelete
  153. [...] A really nice article, on how Facebook ships code. I am not sure if it is 100% correct but even it has some truth in it, it should give other companies a clue about how to treat engineers. [...]

    ReplyDelete
  154. Here's an explanation on what product managers are: http://www.pmhut.com/scrum-product-manager-product-owner-roles-and-responsibilities

    It explains the roles and the responsibilities of the Scrum Product Manager.

    ReplyDelete
  155. With so many corrections one must ask itself: Based on what information did the author come up with this?

    "Oh, they have QA and UnitTests bot nobody uses them". Bam, some other guy writes the exact opposite: "No, we use UnitTests all over the place."

    "If you make many mistakes and get blamed now and then, by time you will eventually get fired" -- "Oh no, nobody here gets fired for mistakes!"

    Seriously, WTF?! Do you actually know *anything* about the internals of FB or did you just want to write something about FB, because, well, to be cool and stuff? There's nothing wrong with comments that add information and clearify some points, but those totally contradicted your statements, rendering this whole article useless.

    Just a big game of bullshit bingo going on here, IMHO.

    PS: No, I don't know anything myself about FB's internals. But at least I'm not going around and writing articles with made up details.

    ReplyDelete
  156. Emm... In this case, "developer driven" sounds like a happy title for a sloppy development process covered by a *very good* operations process. It'd be really interesting to get an idea of the ratio of bounced pushes to successful full-production releases.

    ReplyDelete
  157. Trying again to post a comment... somehow it doesn't seem to work...
    I posted a translation of your article in Japanese.
    http://74k3.blogspot.com/2011/01/read-interesting-article-on-facebook.html

    ReplyDelete
  158. [...] according to this post Facebook simply lets the code live on a microcosm of their users and sees how they react. This is [...]

    ReplyDelete
  159. [...] a tantalizing look into the different processes and practices Facebook uses to build and release code. Granted, it’s a set of notes culled from second-hand information, it’s still an [...]

    ReplyDelete
  160. Okay, it worked with the Website field empty...

    Thanks for a very interesting article. It's great to know how exactly developers supporting such a huge number of users work. I really enjoyed reading the article. As for some topics, I thought it would be more interesting to see them in detail, like how much super-productive they are supposed to be, how is a public shaming specifically, etc.
    Now I wonder how/if the environment have changed since then and if it's working as expected.

    ReplyDelete
  161. [...] http://framethink.wordpress.com/2011/01/17/how-facebook-ships-code/ [...]

    ReplyDelete
  162. FAcebook has to be a very complicated system because it looks really simple... That said think about the amount of functionality it provides, AND that it processes probably bilions of information a day. Thing is a lot of this stuff is going behind the scenes. Have You noticed that news feed only feeds information with only the info about friends You have recently interacted... There is a lot of initiative going up there - search, events, privacy model, platform, internal data warehouse, analytics... soon probably geo features and even more mobile support.
    Do not forget that they have introduced a payment system (it NEEDS coding, security...) and many other stuff. And they claim it's only 50 devs. That's small amount. Very agile though. And very smart, but still it sounds like to organization is very RIGID in its agility.

    HAve You noticed that the only part of information You can actually edit is Your profile (which only You can edit obviously) and small objects like events. This very smart of them (think about propagation of changes...) and makes the problem of advanced rights management and access managment go away.

    I truly admire FB for its model (true also for google and android). And even though I love MS (.net rocks) it can't be that agile in the consumer world (but is very good in business which will probably be it's target market or at least where the money is.)

    ReplyDelete
  163. this is amazing... really great stuff.

    ReplyDelete
  164. First of all, very impressed with the process. Particularly with the OPS metrics/monitoring. Looks like they truly achieve development / release agility, particularly with a huge and complex service like Facebook.

    ReplyDelete
  165. [...] simply did not care about privacy. I am not sure, but if you read Skype Manager Yee Lee’s post on “How Facebook Ships Code” one could argue that the way in which the organization is set up certainly lends itself to some of [...]

    ReplyDelete
  166. [...] 原文在此,看完之后,终于明白为什么优秀的工程师都去了/想去facebook,因为那里是工程师们的天堂。 [...]

    ReplyDelete
  167. [...] Weit gefehlt. Bei Facebook ist jede Art der Entwicklung unheilschwanger. Jeder neue Funktion kann im absoluten Supergau enden. Das belegen zumindest die Veröffentlichungen eines Internetprofis, der Aussagen und Fakten diverser Facebook Mitarbeiter in seinem Blog sammelte. [...]

    ReplyDelete
  168. [...] interesting post about the development process at Facebook. Read it here. Share/Bookmark This entry was written by aft, posted on January 21, 2011 at 8:22 am, filed [...]

    ReplyDelete
  169. [...] at Facebook sounds hard: http://framethink.wordpress.com/2011/01/17/how-facebook-ships-code/ [...]

    ReplyDelete
  170. @rb Project Managers != Product Managers

    Completely different skill sets. PMs are task masters, organisational gurus and often scrum masters. Product managers should own the vision for the product, steer its features backlog, have the right blend of UX/tech and business knowledge, and communicate the road map for the product outside the business and within.

    @jason That's probably because you aren't one and don't know what a good manager is able to do.

    ReplyDelete
  171. [...] How Facebook Ships Code « FrameThink – Frameworks for Thinking People RT @Jason: killer article on How @Facebook/@finkd Ships Code « FrameThink – Frameworks for Thinking People [...]

    ReplyDelete
  172. [...] just read a very interesting blog entry about the dev operations are in facebook here [...]

    ReplyDelete
  173. Hmm ... engineering driven culture good ... public shaming ... um ... what's next? Flogging with a whip?

    And yes, there is no QA based on the edits to this article. Engineers testing their own code is not QA it's just retarded. No wonder every release has bugs and lately has cause the site to go down.

    ReplyDelete
  174. [...] How Facebook Ships Code – 50% of the company is ops/engineering, 90k servers, 9 levels of rollout, no QA. [...]

    ReplyDelete
  175. [...] How Facebook Ships CodeVery interesting insights into Facebook’s developer driven culture. [...]

    ReplyDelete
  176. This article and one first referenced(concerning Mahalo) were INCREDIBLE! Thanks so much for writing it. A developer-driven culture is exactly what my employer needs to become and here is why:

    1. I'm on a project where the business analyst (who owns the project) announced the product on May 3, 2010, but never got completed requirements done until Jan 21, 2011. Our 1st designer quit for personal reasons and our 2nd designer is grudingly moving along...not liking to make frequent UI designs.

    I'm the single developer/architect for a product that may have 10,000 plus users, all the while doing maintenance on other projects. Priority is based on the squeaky wheel theory.

    We have two QA teams at our firm. The fitst QA team has only found 1 bug in all my projects they've tested. The 2nd QA team has tested two other projects : they had one project for 6 weeks and found 0 bugs and another project they had for 4 weeks and found 1.

    In a developer-driven culture we'd have saved on the QA and designers. These two articles really opened my eyes. Now to find such places in Chicago.

    ReplyDelete
  177. [...] How Facebook Ships Code « FrameThink – Frameworks for Thinking People – [...]

    ReplyDelete
  178. [...] How Facebook Ships Code 中文:Facebook是如何管理代碼的 [...]

    ReplyDelete
  179. [...] »自动草稿By Aaron, on 一月 24th, 2011英文原文:How Facebook Ships Code 中文翻译:Facebook 是如何管理代码的全文转载如下: [...]

    ReplyDelete
  180. [...] denkt. Letzte Woche veröffentlichten Frame Think in ihrem Blog den Artikel “How Facebook Ships Code“. Einfach zusammengefasst: Jeder Entwickler kann frei drauflos programmieren, solange er nur [...]

    ReplyDelete
  181. [...] How Facebook ships code. Facebook Management. [...]

    ReplyDelete