@portfast
News & views, serious business.
Wednesday 28 June 2023 - Debian Bookworm
Debian Bookworm was released this month, and with that we are starting to upgrade our systems. Our standard build of the previous release more or less upgrades cleanly, but we had to make the following changes to get rid of a Failed to connect to bus: No such file or directory error which arose when rebooting or during some other actions.
It's caused by the lack of dbus in our standard build, which didn't cause any problems before but now systemd (this isn't a post about whether systemd is awful or not) seems to require interaction with it during certain actions. Fortunately the error seems to be cosmetic, but to make it go away, run the following as root:
apt-get install dbus-user-session
After that, you can start dbus manually, or it should start on reboot and the messages will be gone.
The other problem we encountered is that libapache2-request-perl was dropped. It's a surprise that this module didn't make it into bookworm, we understand that perl may not be the trendy programming language of the week however there is going to be gigabytes of it still holding the internet together. It does at least look to have been picked up again and is in the current testing. As a workaround, we took that version (2.17-3), changed the dependencies to suit and created a bookworm compatible .deb file that has plugged the hole.
Following a recent customer ticket, we would also suggest you not let it replace /etc/default/grub, or if you do, make sure you have GRUB_TERMINAL=serial in there somewhere, otherwise it will try to communicate with a VGA device that's not present and this makes it reboot. If you've already done this, then you can use the recovery console to boot an alternative kernel and fix it.
Wednesday 27 April 2022 - Copper 10Gbase-T transceivers
10 gigabit ethernet is a great technology, offering far more bandwidth than anyone should ever need in the home, it’s most often found running in and between data centres.
It started as a fibre based technology, able to run 10km and beyond over single mode fibre (or multimode over shorter distances if you must).
A relatively recent development has been 10Gbase-T, promising to bring copper cabling over the gigabit mark. I recently purchased a copper 10 gigabit transceiver to see how well it works. The prices for these things are around the €50 mark at the time of writing.
The one I got was identical in size and shape to the old 1 gigabit "copper SFP", some of the earlier 10G-T adapters had significant extra power requirements and protrusions from the SFP socket that might not be welcome if the switch is near the door of the rack or unable to cool the optic.
First test was over a 2 metre Cat 5e patch lead to a HP server with a BCM57810 based card in it. So far so good, it linked at 10Gbps and was able to exchange close to that figure with another server on the same switch connected with a traditional SFP+ optic and fibre cable. Strictly speaking you are meant to use Category 6 for 10G, but this worked fine over the short distance. The switch thinks it is a 10G multimode SFP+.
Moving the test to hard mode was more interesting. I have a "Late 2018" Mac Mini desktop that has a 10g copper port on the back. Between me and the lab switch are a patch panel, a wall socket, a couple of patch leads and about 20 metres of cable in the walls, all this is Cat5e spec, not the Cat6 that 10G on copper demands. This was of course fine for the 100 megabits that I ran over it after I installed it a decade ago but will it run at 100x that speed?
The answer was simply no, it won’t, however it then went and surprised me somewhat as it did sync at 5Gbps.
I was not anticipating any speeds other than 1 or 10 out of this device, so that was unexpected. The switch still saw it as a 10Gbps multimode SFP, however it seems that when it is working at one of these multi-gig speeds, it deals with the difference in speed by just dropping the next packet if one is already “in flight”. This led to real world TCP throughput of only about 1.2Gbps (or 600mbps if I pushed the speed down to 2.5G), however all was not lost. With a 50% shaper applied on egress to the switch port, it managed speeds as measured with iperf3 of 4.7Gbps each way.
So I now have a desktop computer linked to the rest of the network at 5Gbps instead of 1. It's definitely progress, and files do copy more quickly between this machine and a local SAMBA based server running on the machine with the 10G SFP+ connection. It's not life changing, although I might appreciate it more if I ever move into video production.
Addendum: having moved house and cabled the new place entirely with Cat6 throughout, the full 10Gbps is now working through patch panels, sockets and so on.
Tuesday 17 March 2020 - Service implications of COVID-19
"May you live in interesting times" is apparently an apocryphal Chinese curse according to Wikipedia, and currently it seems that we do. Ireland has shut its pubs on St Patrick's day so this is serious.
We're well equipped to weather the current storm - earlier in the year we moved all our equipment from London to a data centre here in Sheffield that we have an investment in and have played a part in building. Therefore we won't be denied access to our equipment unless something very strange and unfortunate should happen like a complete lock down.
This data centre is connected out to the world by lit wave and dark fibre services to data centres in 3 other cities (Leeds, London and Manchester) where we pick up transit and peering from a number of other providers, so as long as we have electricity and failing that, diesel, the servers should be fine.
With regards to staff, there's currently one member of technical staff at the data centre (he insisted, apparently if he tried to work from home he would just end up on his playstation - we salute his honesty at least) and otherwise we've always worked remotely so it's business as usual there too.
We would also extend help to anyone in the area involved in either healthcare or the supply chains around food & equipment - if you need help with your networks at this time we have a warehouse with a load of spares and the knowledge to do something useful with them so feel free to ask.
There is also a google doc of other willing networky people here who may be useful.
Tuesday 5 November 2019 - Automatic DNS health checking
DNS can be tricky if you're not quite sure what you are doing with it, so we've decided to take the guesswork out of it.
Our DNS editor gives you the flexibility to do almost anything that you can do with a server that is hosting DNS zones as text files, however this level of flexibility also allows for the user to end up accidentally breaking something.
Our email & hosting accounts will automatically check that your DNS records are in perfect shape, ensuring that your email gets delivered and your site can be seen by everyone, no specialist knowledge is required.
In order to access the checker, access the "Overview" section in the package administrator. The Fix buttons will automatically deploy updated settings.
Tuesday 8 January 2019 - .eu domain names
This bit is a disclaimer. If EURID still decide to delete your names or if anything else happens as a result of acting upon this text then this is out of our control and we can't do anything about it.
Unfortunately the ongoing fiasco in the UK shows no sign of crystallising into anything we can work with even though there are now only 80 days left until the clock runs out under Article 50, so this bit of speculation (and this is all it is, not advice, legal or otherwise) is the best we can do.
EURID (the bit of the EU that runs the .eu top level domain) has produced a document which is not ambiguous, to summarise, .eu domains are for people and organisations that exist within the EU so if the UK leaves then UK residents no longer have a claim to this name space. Removal of .eu domain names from UK residents will happen by not accepting renewals when they are due. The document can be found here.
If you are UK resident and have .eu domains that you want to keep using after March then one possibility is that you could find a friendly resident of a more stable EU country who can take the domain on.
In practice, nothing should ever come through in the post for one of these domains, but you need to make sure they are a real person and that you have obtained consent, as they will become the legal owners.
Friday 9 March 2018 - Automatic renewal
We are aware that a lot of people like things to just work, there are things you don't want to have to think about like renewing your domain name or email hosting.
With this in mind, we have now made it possible for products to renew themselves. It's easy to configure and just requires that you have a current credit card on your account. We're working on direct debit support also.
You can configure it from the "Payment methods" tab on the left.
It is all completely opt-in, so you won't get surprise billed for things you had bought to try out and forgotten about.
Configure automatic renewal now »
Friday 2 February 2018 - Reboots and OOB console functionality
We recently undertook a reboot of quite a few of our longest lived virtual machines, as a change in the way the kernel handles certain structures internal to our chosen virtualisation technology meant that we couldn't live migrate these to a new host to perform maintenance.
As a bit of background to this, we make extensive use of live migration to peform maintenance on our virtualisation platform without having any customer impact. When we want to do something to a host that involves downtime, we can move all the running virtual machines off it seamlessly with almost no downtime, only a few seconds while the layer 3 route update propagates in our core.
Unfortunately what sometimes happens when one has, and we are going to blow our own trumpets here, virtual machines capable of several year uptimes is that things like boot loaders can be broken by software updates and this can go unnoticed for long periods. With this in mind, we have introduced a feature to the out of band (OOB) console which allows you to boot the VM from a kernel file held on the host, bypassing the inbuilt boot loader on the machine and allowing you to get it started so that the boot loader can be reinstated properly.
We have put together some documentation for this feature here.
Tuesday 2 May 2017 - Two factor authentication
As we take security very seriously here, we've decided to implement two factor authentication using Google Authenticator. We selected this as it uses two open algorithms as laid down in RFCs 4226 and 6238. This means no lock-in, thereby not restricting you to a Google product.
It works by taking a secret that we generate and storing that on your device. When we ask for your 2FA one time password, your device generates a number based on this secret and we compare that to a number that we generate using the same algorithm which then confirms that we have the same secret without actually passing it in the clear.
Once enabled therefore, you need to not only know something (your password) to log in, but to have something too in the form of a device capable of calculating the one time password. With this in mind, please make sure your contact details are up to date in case we should ever have to reset your account if you lose all your 2FA devices, as we will need to prove that you are who you claim to be.
Wednesday 21 December 2016 - A very PoE Christmas
Picture the scene: a freshly acquired christmas tree, various lights and decorations retrieved from their summer hiding places and then suddenly, tragedy. One of the power supply bricks for the lights is missing!
A bodge is soon secured, it turns out that the lights are 24v and so is the office scanner. We have light, and they even look better as a result of being fed from a proper smoothed DC power supply rather than straight from a bridge rectifier.
It was however not to last. There were new contracts to sign, and the scanner was required so its power supply had to be retrieved and freed of the small mound of insulation tape that secured the lights to its connector. As a part of this process I noted that there was an ethernet socket close to the tree, and a Cisco 3560G PoE switch on the other end of it. It crossed my mind that it would be very useful if the lights were 48v and not 24.
Hold on a minute, there are two identical sets of 24v lights. A plan formed and the IEEE 802.3AF specification was downloaded.
First I tested the power consumption of the lights, you have a budget of up to 15.4W (or 12.95W after some cable loss) - they were below 3 watts per set so well within budget.
The second task was persuading the switch to supply power. PoE device detection is performed by pulsing the cable with a low voltage and looking for a resistance of 25kΩ, we didn't have that exactly but found a pair of 12kΩ which were close enough to make the switch see a device. The device is then given a short time by the switch to start consuming power, or the supply is disconnected and it goes back into discovery mode. With the lights attached, it starts drawing power straight away and the switch seems happy, and the switch logs this:
Dec 21 2016 07:10:08.218 GMT: %ILPOWER-7-DETECT: Interface Gi0/14: Power Device detected: IEEE PD Dec 21 2016 07:10:08.772 GMT: %ILPOWER-5-POWER_GRANTED: Interface Gi0/14: Power granted
And this is the circuit:
Yes, really. All of it.
I did wonder if at some point the switch would cut the supply due to the device not attempting any further negotiation, but it seems to be quite happy to supply power indefinitely.
With confirmation that this circuit does actually work, I've soldered it and made it nice with some heat shrink, now it is tucked away behind the tree and is much neater than the previous arrangement of cumbersome power bricks, and a lot more efficient. It turns out those old bricks were wasting more than half of the power they consumed.
But wait, this now means my lights are attached to a network device. A device that can be controlled through SNMP...
Enter... The app.
Well, the page. Calling it an app might be giving it ideas a little above its station, it calls a very simple CGI script on the server which sends an SNMP packet to enable or disable the port.
That's all folks, have a great Christmas, and try not to electrocute yourselves.
Thursday 7 July 2016 - Documentation
Good documentation is what sets apart a product from the rest - it doesn't matter how good a service is if you don't know how to use it.
With this in mind, we are in the process of launching a new documentation area on the site. It will cover everything from adding a DNS record through to our new API which will allow you to integrate our services fully with your workflow via a simple HTTP based API.
You can access the documentation via the new link at the top of the page.
We'll be adding more sections to this page as we go, so expect it to fill out over the coming weeks.
Monday 21 March 2016 - Many new TLDs and old TLD price cut
We now support almost 600 types of top and second level domain name, from .abogado (Spanish for "attorney") to .zone.
In addition to this, we have managed to reduce our costs for acquiring .com, .net and .org, which is reflected in pricing from today.
You can browse the price list and check availability on the Domains page.
Friday 19 February 2016 - HTTP/2
As part of some general software housekeeping, I've upgraded the version of nginx that looks after the main site & control panel to 1.9.10, this version supports the new HTTP/2 protocol. Note that this is different from the other efforts to do the same thing e.g SPDY and is expected to replace those in time.
I thought this announcement would be a lot more interesting than it actually is, however there's nothing to report other than that it "just works".
There are browser plugins that will tell you which of the various next-generation protocols are in use.
Friday 5 February 2016 - Virtual machine price refresh
We've revised the pricing of our virtual machines, it had been a while and they were looking a little outdated.
To this end, we have decided on a completely flexible model where you choose RAM and disk space to suit your application, with a linear scale of £4 per GB of memory, 7p per GB of disk space and £1 for the IPv4 address. IPv6 only is actually something we've been asked for in the past, so why not.
Additionally, if you pre-pay more than a month, we give a discount ranging up to 10% for a year.
As we value our existing customers, whenever we make a price adjustment we will extend your service term automatically to ensure that you are on the best rate.
We're also working on a configuration changer for live virtual machines too, so that you can grow them as required on short notice. Until then if you want to change your configuration, just open a ticket and we will make it so.
Learn more » Configure your VM »
Tuesday 5 January 2016 - Firewalls for virtual machines
We've just launched a new feature, an external firewall that sits in front of your virtual server. By default the policy is to allow all, so don't worry that we are restricting your traffic or ports.
From a technical perspective, this firewall sits between your virtual machine and its default gateway, not on the VM itself so should the VM become compromised, the attacker can't relax the rules.
We've tied all this together with an intuitive interface within the control panel that lets you create, sort and commit rules in seconds.
Thursday 31 December 2015 - MySQL to PostgreSQL migration
From day one, from the first lines of code back in 2005, we have used a MySQL database as the back end for pretty much everything. It has served us well, not lost any data and mostly done what we have told it to. It remains the world's most popular database engine due to its shallow learning curve and extensive documentation.
As our requirements have shifted, for some time now, I've been eyeing up PostgreSQL as an alternative.
Postgres supports some more useful data types, like 'inet' as a great example, and 'json' as a native container for a blob of JSON data which you can build a query on, so we took the time to build a migration plan.
Interesting things you might run in to during such a migration will include:
- user is a reserved word in Postgres but not in MySQL. We have had to rename a lot of variables. We thought about just translating it in the database abstraction layer, but this has the potential to lead to confusion later on so we've taken the 'rip the plaster off' approach.
- MySQL's validation on incoming data is not as strict as it should be. Other than it not having specific data types that will validate, for example JSON or IP address data, it will also do things like accept '', i.e not null data but a 0 length string, into an enum data type where this is not defined as an element of the enum. We had to fix up a fair bit of data, particularly from the early days, before it would import.
- You will spend a long time crafting the perfect data migration script, just to end up running it once and then archiving it under at least a metre of concrete so that you never accidentally run it again and toast your production database.
- Any unusual SQL statements that you have used probably involve custom MySQL syntax somewhere along the line, you'll find these need fixing. If you are combining order by and group by and it doesn't work, then max() might be your friend.
- Postgres does not let you send a floating point number as an update to an integer column.
- Postgres treats " and ' differently.
The unit tests have all passed but the two databases do behave slightly differently, so if you find anything that's not working as expected then open a ticket and we'll fix it.
Friday 9 October 2015 - SSH public key support
I have been asked at various times over the years if we can put peoples SSH public keys in place for virtual machine out of band management.
Historically it was a manual process but great news! - I've automated it.
Start here by uploading your keys and then you'll be able to push them out via the normal VM admin console.
We'll extend this feature in the near future to allow you to push them to new VMs as part of the build process and any other products where they'll be useful.
Monday 24 August 2015 - Windows 95 is 20
Happy 20th birthday, Windows 95.
Although my desktop of choice for probably 19 of those 20 years has been based on Linux, I still have some semi fond memories of Windows 95, so maybe time for a nostalgic moment.
We didn't have a CD-ROM drive back then, so it came to us on no fewer than 13 3.5" floppy disks which were all required, one after the other, as the percent completed bar edged painfully towards 100. Suffice it to say that you did not get the free Weezer music video on the floppy disk version.
We duly installed it on our 486 DX/2 50 over the top of the existing Windows 3.1 installation that had come with it, and rapidly found that our 8MB of memory that 3.1 had positively wallowed in was not adequate for this new vision of the future, at least not if you wanted to run more than a couple of things at once.
I did once manage to get it installed on a computer with a 20 megabyte hard disk by using "doublespace" and a second floppy drive, moving files as they were installed onto some spare 5.25" disks to make space and keep the installer going. The installer itself consumed around 7MB of temporary space that was freed afterwards, so then you could move the files back into place and it would boot. Of course, there was not a lot of space left afterwards to do anything with, so it was not a lot of use, but it did work.
It's hard not to feel spoilt now, as I sit in my office with a tiny computer that has 16GB of memory and an internet connection that could deliver the contents of those 13 floppy disks in a couple of seconds. We will see what the next 20 years will bring.
Wednesday 13 May 2015 - Venom
If you work anywhere near computers then you can't have missed the VENOM bug that has been in the news today.
It's an interesting one as it could allow code execution on a virtual machine's physical host, with the privileges of the emulator. This is something we always anticipated when building this platform, and each of our VMs runs as its own user, in a chroot with no permission to write or execute anything, so we hope this should prove an adequate trap should anyone try and exploit this particular bug.
We are obviously in the process of rolling out a patched qemu binary and where possible live migrating users over to it. We have a slight problem there in that the older VMs are running on qemu-kvm v1, which although it seems it should migrate into a v2 hypervisor, doesn't work terribly reliably and the failure mode is for both sides to crash.
We were planning to roll out v2 across the board slowly over a period of months to allow people to reboot into it in their own time, but as this has forced our hand somewhat, we will need to do these reboots imminently. Since some of the older VMs on the v1 platform will have uptimes of well over 1000 days, a reboot is probably due.
To this end, there's now a bit of code on each host which listens for reboot events and patches the metadata of the VM to start it up with the v2 hypervisor. From the inside, this should be identical to v1. There are instructions on how to connect to the VM's out of band console in the admin pages, but if you have any problems rebooting then give us a call.
Tuesday 31 March 2015 - Time is hard
Taking a side-step from the usual internet related topics, I'm going to share my experience of parsing the TDT in a DVB stream. I feel it warrants a wider audience due to the unforseen complexity of this particular piece of work.
The TDT is a stream in the multiplexes that transport digital TV, via satellite or terrestrial, the format is the same. The function of this particular stream (always within PID 0x14) is to carry the date and time.
The full specification of this field is laid out in EN 300 468, along with many other parts of the DVB stream. To start with, you have 16 bits which carry the date in Modified Julian Date format. It dates back to 1957 and according to Wikipedia was contrived to record the orbit of Sputnik on a 36 bit IBM mainframe, but is not, for me, the strangest bit of the spec.
The part which for me defies all explanation is why they have used binary coded decimal for the hours, minutes and seconds.
Here is a bit of C that parses it into normal Unix seconds-since-1970.
Knowing the time is a fairly fundamental part of most TV functions, so just spare a minute to think of the poor soul who will have thought "This won't take long!", before dropping their sandwich and having a little cry, just so your TV knows what time it is.
Tuesday 20 January 2015 - More bandwidth, hurrah
Due in part to our excellent peering at LONAP, we're pleased to announce that VPS customers will now have extra bandwidth to play with.
Starting with a terabyte on the basic package, we hope this should give you one thing fewer to have to think about when choosing a provider.
Friday 2 January 2015 - A new look for a new year.
You might have noticed some changes to the Portfast web site today, after much tinkering and shifting of bits, the new version is finally ready to go live.
This first iteration of the new site aims only for feature parity with the old one, a lot of the underlying code is the same, as is the database that it speaks to. As well as easing the transition, this lets us roll it back easily if the customer reaction is along the lines of "OH MY EYES" or "what does 'internal server error' mean?". We hope that the reaction will be a little more positive, but you have to plan for every contingency.
There's even a new logo, yes, partly inspired by our Twitter account name. It's all about being sociable.
Under the bonnet there are a bucket load of JSON API calls that you can make to automate every aspect of your account, while possible to use these now, the specification has not been nailed down yet and is likely to change as having deprecated the old site we can now start tinkering with larger parts of the underlying system.
There will be a key based authentication system for the API which will avoid you needing to leave your actual login details in a script, more to follow.
Other useful features on the road map include (but not limited to):
- Two factor authentication
- DNSSEC
- More weird and wonderful top level domains
- IPv6, IPv6 and more IPv6
If you find a problem with it, or just have an idea for something that could work better, we would really appreciate it if you could drop us an email, or a ticket, or if that bit is the bit which is broken, a good old fashioned phone call.
We wish you all a happy 2015!
All prices are exclusive of VAT.
Portfast Ltd :: Registered in England #6061075, 331 Millhouses Lane, Sheffield, S11 9HY