Wikimedia blog

News from the Wikimedia Foundation and about the Wikimedia movement

Posts by Rob Halsell

Server Decommission Donations

At this time we have closed submissions.  We have received well over 100 requests, and will not have enough servers to cover those, let alone more.  Thanks for the submissions!  ~ RobH @ 2011-06-18 @ 10:00 EST

Due to the overwhelming response, we will unfortunately not be able to reply to everyone on an individual basis.  If your organization is selected, you will receive an email from us indicating the approval, as well as shipping information. ~ RobH @ 2011-06-30 @ 13.30 EST

 

Wikimedia Foundation  has  been upgrading and adding new servers to keep up with traffic demand and capacity growth as we always do. Recently, we replaced some of our older servers with faster, higher capacity  and more energy efficient servers.  These  older servers are now decommissioned and will be donated away. Do note  that they are over 3+ years old and are out of warranty.  While we may have placed a lot of demand on them  over the years, they are in fine working condition.

Most systems (but possibly not all) have the following specifications:

  • Dual CPU 2.5 GHz
  • From 3GB to 24GB of RAM, depending on role.
  • Most have 80 GB or larger HDD (some have two hard drives, some drives are 160GB or possibly even 250GB)

If you are interested, please provide the following information in your email to us:

  • Registered non-profit name and information.
  • Your contact information, including email address, phone number, and relationship with requesting non-profit.
  • Information on the non-profit, their charter,  mission  and goals.
  • Shipping address information for a FedEx Ground delivery (i.e., the shipment destination)*
  • How the servers will be used.  (We like to know and share with folks!)

* At this time we regret that we are only able to ship servers to USA based non-profits.  This is due to the cost of shipping and the various exportation laws and taxes that result from shipping internationally.

Please provide as much detail as possible on how you plan to use the servers. For example,   ‘Wikimedia will use these for our sites.’ is pretty vague where as  ‘Wikimedia is the non-profit foundation that runs Wikipedia.  Server donations to us would be used to run our websites that allow access to Wikipedia and its sister projects.’ is much clearer.

If you are not a registered non-profit, your use of the server(s) must be utilized in a fashion that works with or on the projects of the Wikimedia Foundation.  We are not donating these servers to private individuals for personal use.  All requests that are not for use on Wikimedia projects or are not going to a non-profit will be ignored.

By submitting and possibly accepting servers from us, you are granting the Wikimedia Foundation permission to publish details of the donation.  This is normally (but not limited to) a quick blurb about it on our Tech Blog (http://techblog.wikimedia.org).

The Wikimedia Foundation provides no guarantee of the hardware donated in any manner.  Any use of the hardware is not the responsibility of the Wikimedia Foundation.

All requests will be reviewed by our technical team, and they will reply back regarding server availability.  Please keep in mind that these are handled on a low priority schedule, with our normal operations taking precedence.  There may be delays in shipping out your request, or we simply run out of servers.

At this time we have closed submissions.  We have received well over 100 requests, and will not have enough servers to cover those, let alone more.  Thanks for the submissions!  ~ RobH @ 2011-06-18 @ 10:00 EST

Rob Halsell

Wikimedia Operations Engineer

Server Decommissioning Donations

We have been upgrading and adding new servers for capacity increases as we always do, unfortunately, thanks to the more updated tech, some of our older systems just are not worth the space in our racks.  These are servers that Wikimedia has used on the projects for 3+ years, so they are out of warranty.  These are good systems though, and while we may overload them and want replacements, they are fine for many, many non-profits to use.

Most systems (but possibly not all) have the following specifications:

  • Dual CPU 2.5 GHz AMD (some intel, but most AMD in this batch.)
  • 3-4GB RAM Each
  • Most have 80 GB or larger HDD (Some have two hard drives, some drives are 160GB or possibly even 250GB)

Disclaimers: The Wikimedia Foundation does not guarantee the operation or use of these servers in any shape or form.  They are old, some may have dying fans, bad hdd sectors, and the like.  Servers have been wiped of information, and they ran through that, but no promises on function!  Also, most servers have rails, but occasionally one may not, and we do not sort through them for these things.  However, they are standard supermicro servers in most cases, and getting replacement rails is fairly simple.  Some servers are well over 3 years old, we do not just turn off servers when they hit the 3 year mark, we turn them off when they are no longer worth using in any role or function on our cluster in a reliable manner.  In most cases, it is simply the hardware technology has updated to the point that a new server is much faster, and since we demand high performance of our servers, it is worth upgrading for our needs.

Now, last time we opened this up, we got a lot of individuals asking for servers for various uses.  Some requests were not very clear, and overall we ended up with a lot of stuff to sort through to find those eligible for these servers.

We try to only donate servers to other non-profits whose core values are similar or in support of our own.  This means we do not donate them for individual use.   Since these servers were purchased with donations to support Wikimedia, we feel we need to further donate them to other like-minded organizations, since that is how the money for the servers was meant to be spent.  This means that we cannot, in good conscience, donate these servers for profit or personal use to individuals or corporations.

If you would like to receive some of these servers for your NON-PROFIT use, please email servers@wikimedia.org.  TO BE ELIGIBLE YOUR EMAIL MUST INCLUDE THE FOLLOWING:

  • Subject: Server Decommissioning Donations for <NONPROFIT NAME HERE>
  • Your name, contact information, relationship with non-profit requesting the servers
  • Registered non-profit name and information.
  • Information on the non-profit, who they are, what their mission statement and goals are.
  • Shipping address information for a FedEx Ground delivery to where the servers need to go.
  • How the servers will be used.  (We like to know and share with folks!)

Please keep in mind that deciding where these go is pretty tough, so the more detailed you can be in your email is best.  (IE: ‘Wikimedia uses these for our sites.’ is pretty vague where ‘Wikimedia is the non-profit foundation that runs Wikipedia.  Server donations to us would be used to run our websites that allow access to Wikipedia and its sister projects.’ is a lot nicer. ;)  Also, by submitting and possibly accepting servers from us, you are giving us permission to post about it here on our technical blog.

The submission period will remain open no less than two weeks from this posting.

The last time we did this was in September 2009, which took a long while to get sorted out!  We try to do these in multiple, smaller batches, it just easier on operations that way.

Rob Halsell, Operations Engineer

If there are any questions or concerns, anyone can feel free to email me @ the servers@wikimedia.org address, or you can also feel free just to post the questions in the comments.  (Which is even better, avoids duplicate questions!)
Rob Halsell
Systems Administrator

Wikimedia donates servers to deserving non-profits.

Every year, Wikipedia usage goes upward, and every year the technical folks working and volunteering with Wikimedia have to plan, purchase, and implement new servers to keep up to the growing popularity of Wikipedia and its sister projects.  With the advances in computing, running 9 new application servers this year took the load of 36 application servers from 3 years ago.

So when we upgrade, what happens to the old equipment that is too slow for Wikipedia, but not too slow for MANY other non-profits?  We donate them!  These systems were 1U rackmount servers, dual cpu 2.5-3, single core, 2-4GB of RAM, and 2-4 HDD Bays with 1-2 80-250GB HDDs. This year, we have  three non-profits who received our older systems (in alphabetical order): Drupal.org, OpenStreetMap Foundation, and Sugar Labs.

Drupal.org

Drupal is a free software package that allows an individual or a community of users to easily publish, manage and organize a wide variety of content on a website. Tens of thousands of people and organizations are using Drupal to power scores of different web sites.

OpenStreetMap Foundation

The OpenStreetMap Foundation is an international non-profit organisation supporting but not controlling the project. It is dedicated to encouraging the growth, development and distribution of free geospatial data and to providing geospatial data for anybody to use and share.

OpenStreetMap is an open initiative to create and provide free geographic data such as street maps to anyone who wants them.

Sugar Labs

The mission of Sugar Labs® is to produce, distribute, and support the use of the Sugar learning platform; it is a support base and gathering place for the community of educators and developers to create, extend, teach, and learn with the Sugar learning platform.

We hope the recipients of our servers will be able to put them to good use!

Below are some common questions involving Wikimedia and the server donation process:

Q. How can I get some of the decommissioned donation servers?

A. The best place to follow the goings on of our technical team is here, on the Wikimedia Technical Blog.  When we have a batch of servers up for decommissioning and donation, we will announce it on the tech blog, and instructions on how to apply to receive some servers.

Q. Who is eligible to apply for servers?

A. We try to only donate servers to other non-profits whose core values are similar or in support of our own.  This means we do not donate them for individual use.   Since these servers were purchased with donations to support Wikimedia, we feel we need to further donate them to other like-minded organizations, since that is how the money for the servers was meant to be spent.

Q. How often does this happen?

A. Most servers are kept in use by Wikimedia beyond three years.  Many of our servers that we have turned off in this batch are anywhere from 3 to 5 years old.  We only replace them when it makes sense from the technical standpoint to do so.  This means we cannot just say ‘we will do this every X months.’  We try to get the most use out of every server, as they were donated or purchased with donations.  So there is no set date, just keep checking the Wikimedia Technical Blog, when we have more to donate, we will say so there!

Q. I am a student/person/so and so, and I want to learn to develop and do such and such.  Can you send me a server?

A. Sorry, unfortunately it is just not realistic or fair of us to try to sort out which personal use requests for servers are legitimate and which are folks wanting computers for any other reason.  We choose to limit our donations to other like minded non-profit organizations.

Rob Halsell
Systems Administrator

Server Donation Entry Period Ending

Just to let folks know, we have had quite a large interest in our donation of some of our decommissioned servers.  In fact, I have way too many emails!

So to be fair, rather than just stop today, we will stop accepting submissions for this next Monday, September 28th.  That means if you want your proposal/request in the running, you have to have it emailed to servers@wikimedia.org by Midnight GMT this coming Sunday, Sept. 27th.

For ease of reference, here is a copy of the post from the start of this process:

It is that time again.  We have approx 35 servers to donate to a good home.  These are servers that Wikimedia has used on the projects for 3+ years, so they are out of warranty and just not fast enough for us to keep using on the cluster.

The servers will go out to homes for folks who are willing to pay for the freight.  They are as follows:

  • Dual CPU 2.5 GHz AMD
  • 3-4GB RAM Each
  • Most have 80 GB or larger HDD

Disclaimers: The Wikimedia Foundation does not guarantee the operation or use of these servers in any shape or form.  They are old, some may have dying fans, bad hdd sectors, and the like.  Servers have been wiped of information, and they ran through that, but no promises on function!

If you would like to receive some of these servers for your NONPROFIT use, please email servers@wikimedia.org.  Please include in your email how you will be using the servers, and the address they would be shipped to.  We will review all requests and try to fairly pick out where they go.  (Selection process may be refined, but it also may just include throwing darts at a board to break up ties.)

Additions: Due to request, the servers are indeed located in Tampa, FL USA.  Zip code 33602 for shipping purposes.  This means that if you are international, shipping this hardware is really not cost effective for you.  If you want to be in the running still, and are comfortable with personally handing all customs, duties, export, and tax issues, go ahead and email us.

Rob Halsell, Operations Engineer

Correction: Dates were off.

Server Donation Time Again!

It is that time again.  We have approx 35 servers to donate to a good home.  These are servers that Wikimedia has used on the projects for 3+ years, so they are out of warranty and just not fast enough for us to keep using on the cluster.

The servers will go out to homes for folks who are willing to pay for the freight.  They are as follows:

  • Dual CPU 2.5 GHz AMD
  • 3-4GB RAM Each
  • Most have 80 GB or larger HDD

Disclaimers: The Wikimedia Foundation does not guarantee the operation or use of these servers in any shape or form.  They are old, some may have dying fans, bad hdd sectors, and the like.  Servers have been wiped of information, and they ran through that, but no promises on function!

If you would like to receive some of these servers for your NONPROFIT use, please email servers@wikimedia.org.  Please include in your email how you will be using the servers, and the address they would be shipped to.  We will review all requests and try to fairly pick out where they go.  (Selection process may be refined, but it also may just include throwing darts at a board to break up ties.)

Additions: Due to request, the servers are indeed located in Tampa, FL USA.  Zip code 33602 for shipping purposes.

Rob Halsell, Operations Engineer

Presentations from Wikimania and More

Many folks do not know, but we actually try to upload and make available all our presentations.  Presently, you can see a list of them on our Wikitech wiki.  You can follow this link to see them all.

Keep checking back, because the conference isn’t finished!!

PMTPA Router Reboot – Scheduled Downtime (Resolved)

Our primary router for the pmtpa cluster had to be rebooted today at 12:00 GMT.  A line card had died and needed replacing, and the

120px-Gnome-face-sick.svg

system required a reboot for it to fully take effect.  Once that finished, CentralNotice was adding a lot of overhead and had to be disabled for our caching cluster to catch up.  Then the overload caused the primary database master for S3 to overload, and we are in the process of switching database masters to another server.

If all went as planned, this would have been a quick 5 minute router reboot and back online.  Unfortunately, things do not always work smoothly, so what would have been 5 minutes has been awhile.  This post will be updated as more details are resolved.

Update: We have switched database masters successfully and all sites and projects should once again be fully functional as of 14:13 GMT.

Rob Halsell, Operations Engineer

ESAMS Servers not reachable, some EU traffic affected. (Fixed!)

Starting approx. 03:20 GMT, servers in our ESAMS facility began to roll offline one after another.  After some investigation, it appea

rs power is not being supplied to all the servers.  This has resulted in some slow downs for traffic of EU users.

120px-Gnome-face-sick.svg

We have temporarily migrated all traffic to our primary FL datacenter.  Once the servers are

back online in ESAMS, we will be pushing service back to it as well.

Update: The problem has been identified and finally fixed. Traffic has been returned to normal.

The best guess so far is that there was a cooling failure in the datacenter which caused the Sun boxes to shut themselves down.

An update from Leaseweb/Evoswitch is here:  http://noc.leaseweb.com/status.php?i=389

Rob Halsell, Operations Engineer

PDF Export currently down (fixed)

Our PDF export server is presently down.  It had to be rebooted to organize and route some power cables in our racks.  When it powered back on, it is failing to load all software correctly.  We are working on resolving it, I just wanted to post something here on the blog since it is the first place that many people check when they think some service is broken.

Blog Downtime

I am sure that many folks noticed that on the morning of 2009-06-26, techblog.wikimedia.org and blog.wikimedia.org went down.  It turns out that some of the parts of our WordPress installations were compromised.  I do not want to get in to a direct show and tell of what they did, but hopefully we have hardened the installation to the point that it will not occur again.

This is why the blogs exist on their own server, so when things like this happen we can minimize the impact.  The blogs are both up and running now, along with the other services that were affected.  All but techblog was back online before Friday was over, techblog lagged behind until today.  (As techblog was the point of exploit, we got everything else back up first.)  Other affected services were the Open Conference Systems site for Wikimania 2009, as well as our survey software.  Both of those were back online ASAP after the incident and the rest followed after.

Of course, it was hard to get this information out to folks when the blogs were down!  It goes to show how easily using the blogs to get info out has been, since without it we had to scramble to get the information out of other channels.

Thanks to everyone who assisted in the restoration, and also thanks to everyone for their patience while the system was fixed.