Hi! I'm Joris

Newsletter #31 - How I spent 1,200 dollar and two sleepless nights on Cloud costs

Published 12 months ago • 6 min read

Newsletter #31 - How I spent 1,200 dollar and two sleepless nights on Cloud costs

After getting surprisingly good feedback on Product Hunt for Random Airport, I submitted it in an impulsive burst on Show HN on Friday March 31.

The PH story can be found in this previous newsletter.

I put it up around noon, and it gets good traction. At a certain point, I'm number 3 on the homepage. Despite the Product Hunt success, I didn't expect this.

I'm feeling excited, and do my best to reply to the comments and suggestions.

At 4pm, my spouse and I are getting in the car. We're leaving for the beach.

It's a 4 hour drive. She's driving first. I'm entertaining the 10 month old baby on the back seat.

At 5pm I receive an email from Google, and a text message from Citibank. 3,000 THB (~ 100 USD) for Google Cloud has been charged. No context.

I'm slightly confused but not too worried. I take a note to check what this is later that night, when the baby is asleep, I add a couple of other trivial todos.

I tell my spouse there are these stories of people waking up with thousands of dollars of Amazon cloud costs.

At 10pm we have arrived, unpacked, and the baby finally went back to sleep.

I open the laptop.

This was the first time I use Google Maps APIs, and the console is confusing.

It tells me my balance is 20,000 THB (~660 USD). At the same time the underlying "cost table" tabs tell me my balance is 6,000 THB ( ~200 USD). But the "cost breakdown" tab tell me I have something like 14,000 THB in credits. That nicely adds up so I think everything is fine.

But there is a voice in my head ...

When the baby wakes up at 3am, daddy goes to work (after first putting the baby back to sleep).

My balance is now at 29,775 THB (~1,000 USD).

The console tells me it will credit at 30,000 THB. I still don't understand what's happening. The costs on the "cost table" and "costs breakdown" tabs are significantly lower. By now it's clear to me there is a delay on the information, but not how much it is behind. The balance also doesn't update live, and there as well it's unclear how long the delay is.

At 3.45am, I get an email from Google and a text message from Citibank: 30,000 THB has been credited.

It's real, I just paid a 1,000 USD.

I take the site down.

At this point, I'm still thinking this is the cost of my traffic. I have had around 3,000 users at that point in 12 hours.

I go to bed.

And I get up again.

I'm going to reach out to Google, I want to understand this, and see if anything can be done about this.

I start a chat, and after some initial conversation, I get connected to the technical customer service rep. His name is Jesus.

In the hours before, I checked traffic on Google Console and Analytics, I checked the information on my host

But what I didn't do, was check API traffic.

The geocoding API was the culprit, doing 20 times more calls than the Maps api. They should be 1:1 on par!

Jesus tells me I made 273,292 API calls to the Geocoder.

It quickly dawns on me that while building the app, I had an issue with the Maps component which kept re-rendering (it's build in React).

I thought I solved that. Clearly I didn't. The Maps component keeps re-loading, and keeps calling the API.

Jesus tells me to implement restrictions on the API key, either by IP or based on the website. I tell that I will check the code, and solve the issue before putting it back online.

Jesus tells me they will evaluate my case. I'll have a decision in a week.

At 5am, I sign off. Back to bed.

One hour later, 6am, the baby wakes up, as she always does. It's April 1st, the start of a new day :)

The day is fuzzy.

Lack of sleep, intense Thailand heat, and a busy baby keep me distracted. While my subconsious brain seems to be doing a lot of thinking. It also seems to be telling me to get back to the computer.

In the meantime, I'm not on the frontpage of HN anymore, but I'm still #5 on Show HN.

In the evening. At 8pm, when the baby is asleep, I find the problem easily. A clean solution isn't available. I decide to cut the Geocoding API altogether, and upload all the airport coordinates manually to my database. Not fun, but the cleanest solution for now.

Saturday evening 10pm, the site goes live again. And continues to get traffic. It's still #6 on Show HN.

I go to bed, and hope the baby will sleep throughout the night.

I also hope I didn't make a mistake. These Google Cloud emails and Citi text messages shortened my life expectancy.

What I learned and what I'm frustrated about

I learned not to build YOLO api connections when my credit card is attached!

Then again, a thread about huge bills is pinned on the Google Cloud Reddit page.

Clearly I'm not the only one.

And so I'll blame Google as well:

  • For having a billing dashboard with 5 times the same balance information, displayed in a different way, but what is worse, each of them displaying different balances!
  • On top, the information that one should really have, the link to the API usage, is totally absent.
  • Similarly, the email about cloud costs is completely devoid of information. Why are they able to bill so fast, but can't send a warning about massive API usage?
  • And finally, why aren't there easy ways to set limits to cut the usage once it crosses a certain threshold? (I found out later you can set a quota, which is in a completely different section - the Google Maps section - of the dashboard).

Overall the Console is way too bloated. And I thoroughly regret now building it on top of Google Maps, though admittedly:

  • The customer service is great, it's 4am Thailand time on a Sunday morning and I'm speaking to 3 different service reps with a total waiting time of less than 2 minutes. Kudos for that.
  • And two weeks after the blow-up, on Monday April 17, I receive an email that Google will refund me.

The Reddit thread mentioned there seems to be a policy around reimbursing the first time. It was the case for me. My intention is not to try twice 😉.

For comparison. For another project, I use the FlightAware API, which has a clear connection between calls and cost. And mentions the 10 minute delay on the graph.

Though admittedly, I haven't used it in a high stress situation (I hope I won't).

Postscript - 2 days later

Sunday April 2nd, two days after putting the website on Hacker News.

It's 2am. The baby starts crying.

She goes back to sleep easily.

I open my phone and I see a text message from Citi. My heart misses a beat. What happened??

It's Google, they charged me 7,520 THB (~200 USD). It's the regular billing apparently, as Saturday was April 1st, and billing happens on the first of the month.

After I shut the website down, an additional 7,520 THB was incurred. The Console interface is definitely not live. But I'm happy, as it's not an unexpected 30,000 THB again. I go back to sleep.

In the morning, 6am, the first thing I do again, is check my phone. All seems good.

I'll go through the habit of nervously checking my phone a couple more times that day.

I'm afraid additional bills might come, given the delay on the information in the console. Or because I might have made a mistake when fixing the code.

It causes an anxiety I will carry for a couple of days (and nights)

Postscript - 3 days later

I mentioned before Google didn't send any email about increased traffic on my API usage.

3 days later, on Monday April 3rd, 8am, I receive an email from Google that my usage is significantly up in the last 24 hours.

It started peaking more than 70 hours ago ...

Something wrong with the automated emails?

Hi! I'm Joris

This is my weekly braindump. You can read it, but I don't write it for you, I write it for me.

Read more from Hi! I'm Joris

Newsletter #58 - Alea Iacta Est ! I write to document my thinking at that exact moment in time. I wrote the draft for this newsletter last week, in the moment. 10 days ago Monday Toyota announced they wouldn’t extend my contract as the USD JPY exchange rate is making me significantly more expensive Two days later on Wednesday they informed that due to headcount pressure they also won’t be able to convert me to full time if I would want to. That means the Toyota story comes to an end. I’m very...

about 1 month ago • 1 min read

Newsletter #57 - Marathon Again! Every week multiple things trigger me to write a newsletter. Sometimes it's mental block, sometimes it's a deep thought, sometimes it's a memory. But if I don't force myself into a weekly schedule, somehow I don't get to it. Last weekend I ran Tokyo Marathon. There were so many reasons to write: On Friday I was nervous before taking my flight. Writing is the best way to write off the nerves. At the airport I was contemplating how this travel breaks my routine....

about 1 month ago • 2 min read

Newsletter #56 - Deep Pockets! Money I just finished a podcast about an ex-prisoner who did bank loan fraud, acquired millions, got caught, spend around ten years in prison, got released and had to start from nothing. And he said he was miserable when he had the millions, and happy when he had nothing. There are similar stories, the book Wild Swans describes a similar maximum happiness at the moment when they're most poor. And yet, money seems to be the solution to many challenges. Where we...

3 months ago • 1 min read
Share this post