Anchor Web Hosting Blog

LCA day 4 – On freedom

Published January 23rd, 2012 by Barney Desmond

It goes without saying that Linuxconf is all about free software, as in both beer and/or speech. A number of today’s talks focused on freedom, in the context of access to data and code, and the freedom to use software (and hardware) the way you see fit.

We actually had two great keynote talks on freedom, I’d like to step back to yesterday’s talk by Karen Sandler (you can see the talk for yourself on on youtube, which I’d highly recommended). Karen was diagnosed with hypertrophic cardiomyopathy, a heart condition that means she could suddenly die at any time. Thankfully there are treatments available, one of which is a pacemaker.

Being the person she is, she immediately asked “what software does it run?”. Long story short, the manufacturer ended up stonewalling on the issue, refusing to provide code or further details even with an NDA. Noone had ever asked before, and everything was pushed back with assurances that the devices are safe, and that they’re approved by the FDA.

It might seem like a trivial matter, but it’s a big deal if you step back and consider it. This device is implanted in your body to regulate your heart. In the event of cardiac arrest, your life could be 100% dependent on it functioning properly. I think it’s safe to say that failure is unacceptable.

Okay, you say, but they work very well for a lot of people. This is true. But the devices are known to be imperfect – putting aside the issue that they may not function correctly when needed, there are clear concerns regarding malicious access by an attacker. There’s published research for this on both pacemakers and insulin pumps for diabetics.

The hard questions clearly irked a lot of people, including her doctor, who was greatly upset that she’d even be asking such things. The practical concerns did eventually win out (though she was able to get an older, less advanced device), leading to this statement:

I became a cyborg lawyer with proprietary software connected to my heart.

Switching focus to social networking, Bdale Garbee (possibly best known for free beards) has been working on FreedomBox, personal servers for social networking. The immediate need for another social network isn’t obvious – the key here is the storage and control of your own personal information. It’s your data, it should be kept on your terms.

As it stands, your data in Facebook/Google+/FoospaceEtc. could be stored anywhere in the world. For all the privacy policies and statements, you don’t know where that information is, or who really has access to it (think of legal jurisdictions). Designed with tiny “plug computers” in mind, this decentralisation should make it feasible to run your own server from home. Whether Australian internet will ever be up to the job is another matter…

There’s a lot we could go on about but for lack of time. In all, it was a very successful conference: a talk was given, ponies were node’d, a mobile phone was sent towards the stratosphere on party balloons, and Project Horus had their own successful launch. Next year we’re off to Canberra for LCA, hope to see you there!

Tags: ballarat, bdale, freedom, freedombox, LCA2012, linuxconf, open source, pacemaker
Posted in FTW

Leave a comment

Comments

LCA day 3 – High Availability

Published January 20th, 2012 by Barney Desmond

Thursday was more of a “practical” day, with plenty of hands-on hacking. This is nothing new, but nowadays you’re more likely to talk about running a bittorrent client on your bluetooth headset than linux on your toaster. There’s some genuinely awesome, really cool hacks out there (Android and Arduino is where a lot of it’s at), but they’re unlikely to help us give you 99.8% uptime.

Instead, we’ll have a really quick rundown of the high availability (HA) and virtualisation talks, and why it’s a good thing we sent a sysadmin along to them.

Complexity is your biggest enemy when trying to build reliable systems. Complex systems tend to be flaky, and that means they’re unpredictable. Unpredictable systems are bloody hard to support and rely upon. You won’t read this in all the you-beaut cloud services literature, but highly available systems are complex. Really, really complex.

This is all manageable, but it means your staff need to be trained with an intimate understanding of everything, top to bottom. When you’re unfamiliar with it, the HA stack on linux is like the bogeyman. It scares the living daylights out of you, and you try to pretend that if you close your eyes it’ll just go away. This is okay most of the time, but for a company like Anchor it would leave you dependent on a small team of HA gurus when things go wrong.

Thank $DEITY for the High Availability Sprint at LCA. Anchor can train you in The Way Of The Cluster if you so desire, but an enlightenment session from the jedi grandmasters is immeasurably valuable. Knowledge breeds confidence, and these things translate to a more effective sysadmin. If you’re an Anchor customer with an HA system, it means we can support you better, and respond faster when there’s a problem. Everyone wins!

To wrap up, a quick look at the presentation on Ganeti, software for management of a cluster of virtual machines.

We evaluated Ganeti for our needs a couple of years ago as a VM solution, and found that it wasn’t mature enough to really be usable. It’s clearly grown up since then, but I think it might be more interesting to discuss why it’s still no good for us.

Most people can probably look at the featureset and determine whether it’s what they need. Magical on-demand clouds of VMs are the “in thing” at the moment, what aren’t they good for? Well, it turns out they’re not much good for web-hosting.

This really became evident several months ago when we tasked a sysadmin with evaluating the various cloud management products on the market (free or otherwise). It’s kinda disappointing, but the truth is that we don’t need 100 instances of the same machine. We certainly don’t want them to be ephemeral. The other benefits touted by cloudy VMs, such as live migration and replication, are nice but ultimately not that useful for us.

In the end we developed a system that met our real needs, as plain as they are: really fast to deploy, fully automated, customisable, comprehensively supported and monitored.

Tags: ballarat, ganeti, high availability, LCA2012, linuxconf
Posted in FTW

Leave a comment

Comments

LCA day 2

Published January 19th, 2012 by Barney Desmond

Bit of a quiet day today, the highlight was probably the presentations on btrfs and xfs. Btrfs has been developing nicely, and Avi Miller got up to spruik some of the newer features of the filesystem. A bit like ZFS (which isn’t compatible with Linux licensing terms), it pulls in a lot of smarts that are usually the domain of your RAID controller/subsystem. This means more flexibility in how you handle your data, but a lot of new complexity too.

It’s exciting stuff, but we’ll be waiting a bit longer to consider it robust enough to use in production. We’d kill for the integrated snapshotting (great for backups) and data integrity checking (store CRCs with your data) features.

Meanwhile, XFS reports steady progress and positions itself as the filesystem of choice for Really Big systems. Not that anyone would admit to it, but it was clear there was a little bit of rivalry between the two, especially since both talks were back-to-back in the same room.

Dave Chinner talked about how they’ve spent a lot of time working through the metadata performance issues that have caused headaches for scaling-up in the past, and reckons XFS should scale linearly, unlike the competition. Probably not something you’ll lose sleep over when deciding how to format your root filesystem, but definitely important for databases and big filestores.

In lieu of other diversions, let’s have a look at the LeoStick, which was included in the bag of goodies for LCA attendees, alongside the requisite stubby coolers and mousepads.

Unless you’ve been living under a really big rock, the Arduino is the go-to platform for hackers wanting to build embedded systems. This is thanks to ease of programming, fast prototyping, and expansion options (need a thermal probe? fingerprint scanner? CCD camera? there’s probably a single shield module with all of those things). The Leostick is particularly cute in that it comes in USB thumbdrive form-factor. As this is a pre-release board, the more cynical amongst us will note that this is a stroke of marketing genius that should result in some free beta-testing. Heh.

I know a couple of my fair colleagues are handy with a soldering iron; just quietly, this thing may or may not have had something to do with requests from the LCA organisers to stop messing with the exposed USB ports on the electronic door locks around campus.

Tags: ballarat, btrfs, filesystem, LCA2012, linuxconf, xfs
Posted in FTW

Leave a comment

Comments

LCA update, Day 1

Published January 18th, 2012 by Barney Desmond

Anchor’s talk went pretty well by all reports, huzzah!

Actually, it wouldn’t be fair to say it was that easy, so I’ll let the cat out of the bag on this one:

How Anchor's presentation slides for LCA2012 got done in time

Panel 1

T-Rex: Our talk to linux.conf.au got accepted!

Panel 2

{Close-up of T-Rex’s face, he is visibly excited}
T-Rex: It will be AWESOME

Panel 3

{Zoom out to show T-Rex and Dromiceiomimus. T-Rex is about to confidently stomp a tiny house}
Dromiceiomimus: You’ve prepared the talk months in advance, right?
T-Rex: 1337 speakers such as myself need no such preparation!

Panel 4

{Utahraptor replaces Dromiceiomimus in shot, verbally catching T-Rex just as he is about to stomp a tiny woman}
Utahraptor: But what about the slides?

Panel 5

{Now some distance apart, T-Rex and Utahraptor look directly at each other, in tense silence}

Panel 6

T-Rex: Oh uni placement dude?! Can I ask you a favor???

I kid, I kid – they did make the slides themselves, all of them. No uni students were harmed or exploited in the making of this talk.

To wrap up, one talk that covered a topic that doesn’t get much loving was Moving Day: Migrating Big Data from A to B. Mozilla had more than 40TB of data in their crash-reporting system, which demands near 100% uptime, and needed to move it all to a new datacentre – not something to be cowboyed the morning after an all-night bender.

Rigorous planning, automation and testing ensured that everything went smoothly; this talk instilled an idea of how to approach such a mammoth project with confidence.

This is something we handled when Github moved to Rackspace, but Mozilla also added a “post-mortem” phase – even if everything goes well (it did), there are lessons to be learnt from the experience, which stands you in good stead for the next time.

Tags: ballarat, LCA2012, linuxconf, qwantz, slides
Posted in FTW

Leave a comment

Comments

Exciting news from LCA miniconfs

Published January 17th, 2012 by Barney Desmond

Florian Haas gave a talk yesterday at the HA miniconf to present Flashcache, a project that was spawned from Facebook and their desire to squeeze more performance out of their databases.

The basic concept is to use any SSD device as a cache in front of slower rotational media. This is similar to commercial products such as LSI’s Cachecade, but implemented as a linux device-mapper module (so you wouldn’t be able to boot from such a setup, but that’s unlikely to be a real concern).

One of the nice things about Flashcache is that it’s presented as a plain block device. As well as making for a robust and understandable system, a practical upshot of this is that you can also replicate your cache with DRBD. In large HA database setups, this would mitigate a lot of the cache warmup penalty that you suffer after a reboot or failover event.

Flashcache is also fairly configurable, and exposes a lot of stuff through procfs rather than being a black box.

At the moment you have to build it as an out-of-tree module, so of course it’s not the kind of thing we’ll be rushing into production any time soon. Based on what we’ve seen in the past, I reckon there’s a good chance we’ll see Flashcache in mainline in a year or two if there’s a concerted push on development.

Tags: device-mapper, flashcache, high availability, LCA2012, linuxconf, miniconf
Posted in WTF

Leave a comment

Comments

It came from beneath the raised floor

Published January 17th, 2012 by Barney Desmond

Yes, it’s another post about datacentre horrors. I know what you’re thinking: “Yeah yeah, I’ve seen the one about the cabling“.

Yeah well I used to be a datacentre technician like you, then I took a PCI-slot shiv in the knee.

Tags: cabling, datacentre, horror
Posted in WTF

Leave a comment

Comments

Anchor speaking at LCA2012, come listen!

Published January 16th, 2012 by Barney Desmond

I think the title sums it up nicely. If you needed further incentive to come along, I would proudly inform you that my esteemed colleagues Messrs David Basden and Chris Collins will be discussing the finer points of the automated production of heterogeneous server systems. Activities will commence tomorrow (Tuesday) at half-past-ten in room C001, following the completion of elevenses.

In all seriousness, we do hope you’ll come along if you’re attending Linuxconf and this tickles your fancy:
Any monkey can build the same server over and over again reliably.
But what if you need reliable server builds, and every single one is a little different?

If you’d like a little more in-depth detail, the LCA website has a copy of the abstract for the talk.

In addition to presenting, Anchor is sponsoring LCA and has sent a crack team of DevOperatives to look after things. Come and say Hi if you spot us.

Tags: ballarat, LCA2012, linuxconf
Posted in FTW

Leave a comment

Comments

Your Magento store + Anchor = ?

Published January 13th, 2012 by Barney Desmond

A little bit of horn-blowing, the correct answer is of course “a winning combination”.

We often find ourselves bothered by PHP instead of being hot-and-bothered, but Magento is a pretty well-engineered app. It’s got solid documentation (a godsend), and while it’s very resource intensive if you’re a $5-a-month hosting customer, it’s clear they’ve given a lot of thought to scalability for running a serious online shop.

Scalability? Yes please! If you’re interested in that sort of thing, we recently published a little case study about our friends at Games Paradise, and how we helped them gear up for the Christmas season.

Feel free to get in touch if you have any questions, or want to know more about what we do.

Tags: case study, games paradise, magento, performance, scalability
Posted in FTW

Leave a comment

Comments

Ya gotta admire the chutzpah…

Published December 17th, 2011 by Davy Jones

It’s no secret that here at Anchor, we’re not huge fans of the level of support you get from most commercial software vendors. But a recent incident with a certain vendor of crappy hosting management control panels really took the cake…

It all began, as these things do, on a sunny spring morn. The ticket came in, saying “the control panel says our licence is invalid or expired, even though we paid for a new licence a couple of months ago”. As this tends to cause customer-facing outages, it was a fairly important problem that needed fixing.

(Sidenote: Is it really such a clever idea to run a piece of software that has a feature that is deliberately designed and intended to stop the software from working at the deranged whim of the monkeys who sold it to you? I think not)

Digging into the problem, we could find no obvious cause of the fault — firewall open, packets flowing, manual renewal of the licence via the little button in the web UI seemed to work… all very strange.

As the problem adversely effected the customer’s ability to continue to provide money to the vendor, we thought the vendor might be somewhat keen to help rectify the problem, so as to ensure the ongoing supply of said money. So, we contacted their support department.

“You don’t have an extended super-dooper-bend-over-and-take-it support plan; please pay us $90″, replied the support department, with ‘nary a “how do you do” to soften the blow.

“But wait, we’re trying to ensure the customer can continue to pay you money!”, we replied, on the assumption that the support drone on the other end of the e-mail program was just functionally illiterate (isn’t it great the level of service you get for your money)

“We know. We don’t care. Pay up.” was the curt reply.

Well, doesn’t that just obtain the baked goods. In order to get assistance with paying them money, the vendor wants us to pay yet more money. The logic defies all attempts at analysis or explanation.

Posted in WTF

Leave a comment

Comments

Blog

LCA day 4 – On freedom

LCA day 3 – High Availability

LCA day 2

LCA update, Day 1

Exciting news from LCA miniconfs

It came from beneath the raised floor

Anchor speaking at LCA2012, come listen!

Your Magento store + Anchor = ?

Ya gotta admire the chutzpah…

Archives

Blogroll

Recent Posts

Meta