This isn’t a Technology story about the dangers of AI, this is a Risk Management story about the dangers of overconfidence.
For the last week, we’ve been inundated with hot takes about how “Claude deleted PocketOS production database”.
Of course Claude deleted their entire production database (or something similarly damaging), it was just a matter of time. The way this is being discussed misses the point entirely.
Laypeople don’t use AI to plan (then build) bridges or planes or power plants. They require Civil, Mechanical, and Electrical Engineering degrees respectively.
And They Know This.
Let’s apply this same line of thinking to the sad and hilarious overconfidence that allows laypeople to think “Vibe Coding” is the same as “Vibe Engineering”:
In this example, practically any Software Engineer (sure, let’s exclude the bottom 5%) would look at the agent Permissions that allow it to delete a prod database (at all, let alone without human oversight) and flag that as a risk, plus the 3 month Recovery Point Objective and flag that as a risk too.
Don’t get me wrong, yesterday at Blackbird Sunrise I strongly encouraged at least half a dozen pre-MVP aspiring founders to immediately Vibe Code their Prototype and get it in front of their ICP ASAP.
After confirming they weren’t building a product that contained data that was Financial, Health, Defense, or Children, I also encouraged them to launch their MVP and see if they can get to first revenues too.
I then shared with them that this is safe because there’s no risk if there’s no users or revenue, and rapidly after those things changing, they should be speaking to someone with (drumroll), some Software Engineering experience to help them with the next stage of their journey.
Is a lot of the subsequent advice or support they’ll receive pretty basic stuff? Yes.
Did they know to ask about this basic stuff without the gentle, supportive prompting (no pun intended) by the Engineer they engaged?
Not in the case of PocketOS, evidently.
Reach out to an Engineer, we’re here to help folks 👌