Ruining Your Keyspace For Fun And Profit
Oh, the memories.
One of the early design decisions at CoverMyMeds was to generate alphanumeric keys for each Prior Auth, to ease communication about them. We figured trying to get pharmacists to read UUIDs to folks over the phone was a losing proposition. Because it was early, we just slapped together a quick “generate key, check for existence, loop back and generate again if used” routine and shipped it.
As seems obvious in retrospect, we eventaully also generated a key that included the f-slur. Luckily, it was sent to an early partner who found this hilarious instead of wildly offensive. Having dodged a company-ending bullets – not the first, for damn sure not the last – we brainstormed a list of offensive words that could be generated, and denylisted them.
I mean, that’s usually the end of the story, but this one has a fun addendum. Our “PA Key” keyspace, which was previously limited to “all the possible combinations of letters/numbers in our strict format,” got even smaller after we implemented the denylist. That, plus some new partnerships that turned into gangbusters growth, meant that we had nearly no time between the “hey we’re almost out of keys to use for PAs” alarm, and actually being completely out of keys.
Remember, our key generation routine now looked like “generate key, discard and retry if on denylist, discard and retry if already used.” This happened inline during the POST request to create a new Prior Auth, which meant all of our apache threads across all our production servers were spinning, constantly regenerating keys. We’d DOSed ourselves. And it took us a good while to figure out the cause.
Lessons learned: maybe don’t block the inital database CREATE waiting for keyspace to free up. Or maybe it was to just not have a feasibly bounded keyspace in the first place? I dunno, we just changed our key format to give us more keyspace and set up more alarms.