Pushing the whole company into the past on purpose

Pushing the whole company into the past on purpose

Every six months or so, this neat group called the International Earth
Rotation Service issues a directive on whether there will be a leap
second inserted at the end of that six month period. You usually find
out at the beginning of January or the beginning of July, and thus would
have a leap second event at the end of June or December, respectively.

Ten years ago, in January 2015, they announced a leap second would be
added at the end of June 2015. The last one had been three years prior,
and when it happened, it screwed things up pretty bad for the cat
picture factory. They hit kernel problems, userspace problems, and
worse.

This time, I was working there, and decided there would not be a repeat.
The entire company’s time infrastructure would be adjusted so it would
simply slow down for about 20 hours before the event, and so it would
become a whole second “slow” relative to the rest of the world. Then at
midnight UTC, the rest of the world would go 58, 59, 60, 0, and we’d go
57, 58, 59, 0, and then we’d be in lock-step again.

So how do you do something like this? Well, you have to get yourself
into a position where you can add a “lie” to the time standard. This
company had a handful of these devices which had a satellite receiver
for GPS on one side and an Ethernet port for NTP on the other with a
decent little clock on the inside. I just had to get between those and
everyone else so they would receive my adjusted time scale for the
duration, then we could switch back when things were ready.

This is the whole “leap smearing” thing that you might have heard of if
you run in “time nut” circles. Someone else came up with it and they
had only published their formula for computing the lie over a spread of
time. The rest of it was “left as an exercise for the reader”, so to
speak.

Work like this benefits from being highly visible, so I bought a pair of
broadcast-studio style clocks which spoke NTP over Ethernet and
installed them on my desk. One of them was pointed at the usual
GPS->NTP infrastructure, and the other was pointed at the ntp servers
running my hacked-up code which could have “lies” injected.

I’d start up a test and watch them drift apart. At first, you can’t
even tell, but after a couple of hours, you get to where one subtly
updates just a bit before the other one. You can even see it in
pictures: parts of one light up before the other.

Then at the end of the ramp, they’re a full second apart, but they’re
still updating at the same time. It’s just that one goes from 39 to 40
when the other goes from 40 to 41.

Back and forth I went with my test clocks, test systems, and a handful
of guinea pig boxes that volunteered to subscribe to the hacked-up time
standard during these tests. We had to find a rate-of-change that would
be accepted by the ntp daemons all over the fleet. There’s only so much
of a change you can introduce to the rate of change itself, and that
meant a lot of careful experimentation to find out just what would work.

We ended up with something like 20 hours to smear off a single second.

The end of June approached, and it was time to do a full-scale test. I
wanted to be sure that we could survive being a second out of whack
without having the confounding factor of the whole rest of the world
simultaneously dealing with their own leap second stuff. We needed to
know if we’d be okay, and the only way to know was to smear it off, hold
a bit to see if anything happened, then *smear it back on*.

This is probably the first anyone outside the company’s heard of it, but
about a week before, I smeared off a whole second and left the ENTIRE
company’s infra (laptops and all) running a second slow relative to the
rest of the world. Then we stayed there for a couple of hours if I
remember correctly, and then went forward again and caught back up.

A week later, we did it for real and it just worked.

So, yes, in June 2015, I slowed down the whole company by a second.

Of course, here it is ten years later, and the guy in charge just sent
it back fifty years. Way to upstage me, dude.

Keep your files stored safely and securely with the SanDisk 2TB Extreme Portable SSD. With over 69,505 ratings and an impressive 4.6 out of 5 stars, this product has been purchased over 8K+ times in the past month. At only $129.99, this Amazon’s Choice product is a must-have for secure file storage.

Help keep private content private with the included password protection featuring 256-bit AES hardware encryption. Order now for just $129.99 on Amazon!