Wikimedia blog

News from the Wikimedia Foundation and about the Wikimedia movement

Hardware

Server Decommission Donations

At this time we have closed submissions.  We have received well over 100 requests, and will not have enough servers to cover those, let alone more.  Thanks for the submissions!  ~ RobH @ 2011-06-18 @ 10:00 EST

Due to the overwhelming response, we will unfortunately not be able to reply to everyone on an individual basis.  If your organization is selected, you will receive an email from us indicating the approval, as well as shipping information. ~ RobH @ 2011-06-30 @ 13.30 EST

 

Wikimedia Foundation  has  been upgrading and adding new servers to keep up with traffic demand and capacity growth as we always do. Recently, we replaced some of our older servers with faster, higher capacity  and more energy efficient servers.  These  older servers are now decommissioned and will be donated away. Do note  that they are over 3+ years old and are out of warranty.  While we may have placed a lot of demand on them  over the years, they are in fine working condition.

Most systems (but possibly not all) have the following specifications:

  • Dual CPU 2.5 GHz
  • From 3GB to 24GB of RAM, depending on role.
  • Most have 80 GB or larger HDD (some have two hard drives, some drives are 160GB or possibly even 250GB)

If you are interested, please provide the following information in your email to us:

  • Registered non-profit name and information.
  • Your contact information, including email address, phone number, and relationship with requesting non-profit.
  • Information on the non-profit, their charter,  mission  and goals.
  • Shipping address information for a FedEx Ground delivery (i.e., the shipment destination)*
  • How the servers will be used.  (We like to know and share with folks!)

* At this time we regret that we are only able to ship servers to USA based non-profits.  This is due to the cost of shipping and the various exportation laws and taxes that result from shipping internationally.

Please provide as much detail as possible on how you plan to use the servers. For example,   ‘Wikimedia will use these for our sites.’ is pretty vague where as  ‘Wikimedia is the non-profit foundation that runs Wikipedia.  Server donations to us would be used to run our websites that allow access to Wikipedia and its sister projects.’ is much clearer.

If you are not a registered non-profit, your use of the server(s) must be utilized in a fashion that works with or on the projects of the Wikimedia Foundation.  We are not donating these servers to private individuals for personal use.  All requests that are not for use on Wikimedia projects or are not going to a non-profit will be ignored.

By submitting and possibly accepting servers from us, you are granting the Wikimedia Foundation permission to publish details of the donation.  This is normally (but not limited to) a quick blurb about it on our Tech Blog (http://techblog.wikimedia.org).

The Wikimedia Foundation provides no guarantee of the hardware donated in any manner.  Any use of the hardware is not the responsibility of the Wikimedia Foundation.

All requests will be reviewed by our technical team, and they will reply back regarding server availability.  Please keep in mind that these are handled on a low priority schedule, with our normal operations taking precedence.  There may be delays in shipping out your request, or we simply run out of servers.

At this time we have closed submissions.  We have received well over 100 requests, and will not have enough servers to cover those, let alone more.  Thanks for the submissions!  ~ RobH @ 2011-06-18 @ 10:00 EST

Rob Halsell

Wikimedia Operations Engineer

Server Decommissioning Donations

We have been upgrading and adding new servers for capacity increases as we always do, unfortunately, thanks to the more updated tech, some of our older systems just are not worth the space in our racks.  These are servers that Wikimedia has used on the projects for 3+ years, so they are out of warranty.  These are good systems though, and while we may overload them and want replacements, they are fine for many, many non-profits to use.

Most systems (but possibly not all) have the following specifications:

  • Dual CPU 2.5 GHz AMD (some intel, but most AMD in this batch.)
  • 3-4GB RAM Each
  • Most have 80 GB or larger HDD (Some have two hard drives, some drives are 160GB or possibly even 250GB)

Disclaimers: The Wikimedia Foundation does not guarantee the operation or use of these servers in any shape or form.  They are old, some may have dying fans, bad hdd sectors, and the like.  Servers have been wiped of information, and they ran through that, but no promises on function!  Also, most servers have rails, but occasionally one may not, and we do not sort through them for these things.  However, they are standard supermicro servers in most cases, and getting replacement rails is fairly simple.  Some servers are well over 3 years old, we do not just turn off servers when they hit the 3 year mark, we turn them off when they are no longer worth using in any role or function on our cluster in a reliable manner.  In most cases, it is simply the hardware technology has updated to the point that a new server is much faster, and since we demand high performance of our servers, it is worth upgrading for our needs.

Now, last time we opened this up, we got a lot of individuals asking for servers for various uses.  Some requests were not very clear, and overall we ended up with a lot of stuff to sort through to find those eligible for these servers.

We try to only donate servers to other non-profits whose core values are similar or in support of our own.  This means we do not donate them for individual use.   Since these servers were purchased with donations to support Wikimedia, we feel we need to further donate them to other like-minded organizations, since that is how the money for the servers was meant to be spent.  This means that we cannot, in good conscience, donate these servers for profit or personal use to individuals or corporations.

If you would like to receive some of these servers for your NON-PROFIT use, please email servers@wikimedia.org.  TO BE ELIGIBLE YOUR EMAIL MUST INCLUDE THE FOLLOWING:

  • Subject: Server Decommissioning Donations for <NONPROFIT NAME HERE>
  • Your name, contact information, relationship with non-profit requesting the servers
  • Registered non-profit name and information.
  • Information on the non-profit, who they are, what their mission statement and goals are.
  • Shipping address information for a FedEx Ground delivery to where the servers need to go.
  • How the servers will be used.  (We like to know and share with folks!)

Please keep in mind that deciding where these go is pretty tough, so the more detailed you can be in your email is best.  (IE: ‘Wikimedia uses these for our sites.’ is pretty vague where ‘Wikimedia is the non-profit foundation that runs Wikipedia.  Server donations to us would be used to run our websites that allow access to Wikipedia and its sister projects.’ is a lot nicer. ;)  Also, by submitting and possibly accepting servers from us, you are giving us permission to post about it here on our technical blog.

The submission period will remain open no less than two weeks from this posting.

The last time we did this was in September 2009, which took a long while to get sorted out!  We try to do these in multiple, smaller batches, it just easier on operations that way.

Rob Halsell, Operations Engineer

If there are any questions or concerns, anyone can feel free to email me @ the servers@wikimedia.org address, or you can also feel free just to post the questions in the comments.  (Which is even better, avoids duplicate questions!)
Rob Halsell
Systems Administrator

Wikimedia donates servers to deserving non-profits.

Every year, Wikipedia usage goes upward, and every year the technical folks working and volunteering with Wikimedia have to plan, purchase, and implement new servers to keep up to the growing popularity of Wikipedia and its sister projects.  With the advances in computing, running 9 new application servers this year took the load of 36 application servers from 3 years ago.

So when we upgrade, what happens to the old equipment that is too slow for Wikipedia, but not too slow for MANY other non-profits?  We donate them!  These systems were 1U rackmount servers, dual cpu 2.5-3, single core, 2-4GB of RAM, and 2-4 HDD Bays with 1-2 80-250GB HDDs. This year, we have  three non-profits who received our older systems (in alphabetical order): Drupal.org, OpenStreetMap Foundation, and Sugar Labs.

Drupal.org

Drupal is a free software package that allows an individual or a community of users to easily publish, manage and organize a wide variety of content on a website. Tens of thousands of people and organizations are using Drupal to power scores of different web sites.

OpenStreetMap Foundation

The OpenStreetMap Foundation is an international non-profit organisation supporting but not controlling the project. It is dedicated to encouraging the growth, development and distribution of free geospatial data and to providing geospatial data for anybody to use and share.

OpenStreetMap is an open initiative to create and provide free geographic data such as street maps to anyone who wants them.

Sugar Labs

The mission of Sugar Labs® is to produce, distribute, and support the use of the Sugar learning platform; it is a support base and gathering place for the community of educators and developers to create, extend, teach, and learn with the Sugar learning platform.

We hope the recipients of our servers will be able to put them to good use!

Below are some common questions involving Wikimedia and the server donation process:

Q. How can I get some of the decommissioned donation servers?

A. The best place to follow the goings on of our technical team is here, on the Wikimedia Technical Blog.  When we have a batch of servers up for decommissioning and donation, we will announce it on the tech blog, and instructions on how to apply to receive some servers.

Q. Who is eligible to apply for servers?

A. We try to only donate servers to other non-profits whose core values are similar or in support of our own.  This means we do not donate them for individual use.   Since these servers were purchased with donations to support Wikimedia, we feel we need to further donate them to other like-minded organizations, since that is how the money for the servers was meant to be spent.

Q. How often does this happen?

A. Most servers are kept in use by Wikimedia beyond three years.  Many of our servers that we have turned off in this batch are anywhere from 3 to 5 years old.  We only replace them when it makes sense from the technical standpoint to do so.  This means we cannot just say ‘we will do this every X months.’  We try to get the most use out of every server, as they were donated or purchased with donations.  So there is no set date, just keep checking the Wikimedia Technical Blog, when we have more to donate, we will say so there!

Q. I am a student/person/so and so, and I want to learn to develop and do such and such.  Can you send me a server?

A. Sorry, unfortunately it is just not realistic or fair of us to try to sort out which personal use requests for servers are legitimate and which are folks wanting computers for any other reason.  We choose to limit our donations to other like minded non-profit organizations.

Rob Halsell
Systems Administrator

Server Donation Entry Period Ending

Just to let folks know, we have had quite a large interest in our donation of some of our decommissioned servers.  In fact, I have way too many emails!

So to be fair, rather than just stop today, we will stop accepting submissions for this next Monday, September 28th.  That means if you want your proposal/request in the running, you have to have it emailed to servers@wikimedia.org by Midnight GMT this coming Sunday, Sept. 27th.

For ease of reference, here is a copy of the post from the start of this process:

It is that time again.  We have approx 35 servers to donate to a good home.  These are servers that Wikimedia has used on the projects for 3+ years, so they are out of warranty and just not fast enough for us to keep using on the cluster.

The servers will go out to homes for folks who are willing to pay for the freight.  They are as follows:

  • Dual CPU 2.5 GHz AMD
  • 3-4GB RAM Each
  • Most have 80 GB or larger HDD

Disclaimers: The Wikimedia Foundation does not guarantee the operation or use of these servers in any shape or form.  They are old, some may have dying fans, bad hdd sectors, and the like.  Servers have been wiped of information, and they ran through that, but no promises on function!

If you would like to receive some of these servers for your NONPROFIT use, please email servers@wikimedia.org.  Please include in your email how you will be using the servers, and the address they would be shipped to.  We will review all requests and try to fairly pick out where they go.  (Selection process may be refined, but it also may just include throwing darts at a board to break up ties.)

Additions: Due to request, the servers are indeed located in Tampa, FL USA.  Zip code 33602 for shipping purposes.  This means that if you are international, shipping this hardware is really not cost effective for you.  If you want to be in the running still, and are comfortable with personally handing all customs, duties, export, and tax issues, go ahead and email us.

Rob Halsell, Operations Engineer

Correction: Dates were off.

Server Donation Time Again!

It is that time again.  We have approx 35 servers to donate to a good home.  These are servers that Wikimedia has used on the projects for 3+ years, so they are out of warranty and just not fast enough for us to keep using on the cluster.

The servers will go out to homes for folks who are willing to pay for the freight.  They are as follows:

  • Dual CPU 2.5 GHz AMD
  • 3-4GB RAM Each
  • Most have 80 GB or larger HDD

Disclaimers: The Wikimedia Foundation does not guarantee the operation or use of these servers in any shape or form.  They are old, some may have dying fans, bad hdd sectors, and the like.  Servers have been wiped of information, and they ran through that, but no promises on function!

If you would like to receive some of these servers for your NONPROFIT use, please email servers@wikimedia.org.  Please include in your email how you will be using the servers, and the address they would be shipped to.  We will review all requests and try to fairly pick out where they go.  (Selection process may be refined, but it also may just include throwing darts at a board to break up ties.)

Additions: Due to request, the servers are indeed located in Tampa, FL USA.  Zip code 33602 for shipping purposes.

Rob Halsell, Operations Engineer

PMTPA Router Reboot – Scheduled Downtime (Resolved)

Our primary router for the pmtpa cluster had to be rebooted today at 12:00 GMT.  A line card had died and needed replacing, and the

120px-Gnome-face-sick.svg

system required a reboot for it to fully take effect.  Once that finished, CentralNotice was adding a lot of overhead and had to be disabled for our caching cluster to catch up.  Then the overload caused the primary database master for S3 to overload, and we are in the process of switching database masters to another server.

If all went as planned, this would have been a quick 5 minute router reboot and back online.  Unfortunately, things do not always work smoothly, so what would have been 5 minutes has been awhile.  This post will be updated as more details are resolved.

Update: We have switched database masters successfully and all sites and projects should once again be fully functional as of 14:13 GMT.

Rob Halsell, Operations Engineer

Intermittent media server load problems

pokey-file-serverWe’ve been seeing some general slowdowns in our image and media file serving recently, including some instances in the last couple days where the sites as a whole have been affected to the point of extreme slowness or temporary inaccessibility.

Domas believes this is related to this reported problem with NFS performance when ZFS snapshots are active. We’ve had some luck so far with it improving after dropping older snapshots (possibly along with restarting NFS and temporarily disabling the image scaler servers to give it a little breathing room to reset).

We’ve been planning for some time to redo the way we access our media files internally which can help reduce the impact on the rest of the site when load problems on the file servers occur, but we might also be able to spread out the load among multiple servers to improve things even more.

Updates will come as we get things back on track…

Update 2009-07-15: We’re temporarily shutting off uploads while we apply the ZFS fix patch and reboot the main file server. You may see some missing images or funky error messages for a little bit, but the sites should otherwise continue working normally until the file server is back up.

Update 2: Server is patched and uploads are back online. This should resolve our performance problems while we continue rearranging the upload servers to be more future-proof.

Brion Vibber, Lead Software Architect

WMF needs additional datacenter space

The Wikimedia Foundation is looking into the option of expanding into a new datacenter.  Currently the plans are tentative, but are expected to become much firmer once discussions with various Datacenter Providers takes place.

Currently, the servers for the projects reside in Tampa, Florida, USA, and in Amsterdam, Netherlands.  We actually have moved the servers recently in Amsterdam.  Now the time has come to move/expand in the US.  We are looking at moving to an area OUTSIDE of Florida, where every single hurricane season is not the cause of distress.

We are currently looking in the Virginia and DC areas, but are not adverse to other areas given the space/power/transit issues.  I have already been in contact with a number of vendors, but that doesn’t mean I do not want more options.

Things we require:

  • We are looking for Datacenters that offer co-location services with 24/7 access.
  • We also require racks have both primary and redundant power drops, from different feeds and circuits.
  • The drops also need to be 3phase 208V power.
  • Offers a low cost out of band access for our mangement network ONLY (no production traffic.)
  • Some kind of NOC in residence in the event of ‘horrible end of the world’ happenings and we need remote hands.  (We have LOM and remote reboot capabilities, but having a NOC is never horrible.)

Any interested sales folks at a datacenter can email me rob at wikimedia dot org.  Put ‘Datacenter Relocation Project’ in the subject so I am sure I see it!

Also, any folks out there who have decent recommendations, let me have em!

Pretty Servers

Since we now have our own blog, we can do neat stuff like point out that we have photos of one of our new datacenter deployments. They are tagged on Commons with the Category of Wikimedia Servers. However, if you just want to see the new photos you can do that here. Keep in mind all these are possible due to the generous donations of our readers!

Rob Halsell, Operations Engineer