Newsletter #31 - How I spent 1,200 dollar and two sleepless nights on Cloud costs

After getting surprisingly good feedback on Product Hunt for Random Airport, I submitted it in an impulsive burst on Show HN on Friday March 31.

The PH story can be found in this previous newsletter.

I put it up around noon, and it gets good traction. At a certain point, I'm number 3 on the homepage. Despite the Product Hunt success, I didn't expect this.

I'm feeling excited, and do my best to reply to the comments and suggestions.

At 4pm, my spouse and I are getting in the car. We're leaving for the beach.

It's a 4 hour drive. She's driving first. I'm entertaining the 10 month old baby on the back seat.

At 5pm I receive an email from Google, and a text message from Citibank. 3,000 THB (~ 100 USD) for Google Cloud has been charged. No context.

​

I'm slightly confused but not too worried. I take a note to check what this is later that night, when the baby is asleep, I add a couple of other trivial todos.

I tell my spouse there are these stories of people waking up with thousands of dollars of Amazon cloud costs.

At 10pm we have arrived, unpacked, and the baby finally went back to sleep.

I open the laptop.

This was the first time I use Google Maps APIs, and the console is confusing.

It tells me my balance is 20,000 THB (~660 USD). At the same time the underlying "cost table" tabs tell me my balance is 6,000 THB ( ~200 USD). But the "cost breakdown" tab tell me I have something like 14,000 THB in credits. That nicely adds up so I think everything is fine.

But there is a voice in my head ...

When the baby wakes up at 3am, daddy goes to work (after first putting the baby back to sleep).

My balance is now at 29,775 THB (~1,000 USD).

The console tells me it will credit at 30,000 THB. I still don't understand what's happening. The costs on the "cost table" and "costs breakdown" tabs are significantly lower. By now it's clear to me there is a delay on the information, but not how much it is behind. The balance also doesn't update live, and there as well it's unclear how long the delay is.

At 3.45am, I get an email from Google and a text message from Citibank: 30,000 THB has been credited.

It's real, I just paid a 1,000 USD.

I take the site down.

​

​

At this point, I'm still thinking this is the cost of my traffic. I have had around 3,000 users at that point in 12 hours.

​

​

I go to bed.

And I get up again.

I'm going to reach out to Google, I want to understand this, and see if anything can be done about this.

I start a chat, and after some initial conversation, I get connected to the technical customer service rep. His name is Jesus.

In the hours before, I checked traffic on Google Console and Analytics, I checked the information on my host Render.com.

But what I didn't do, was check API traffic.

The geocoding API was the culprit, doing 20 times more calls than the Maps api. They should be 1:1 on par!

Jesus tells me I made 273,292 API calls to the Geocoder.

It quickly dawns on me that while building the app, I had an issue with the Maps component which kept re-rendering (it's build in React).

I thought I solved that. Clearly I didn't. The Maps component keeps re-loading, and keeps calling the API.

Jesus tells me to implement restrictions on the API key, either by IP or based on the website. I tell that I will check the code, and solve the issue before putting it back online.

Jesus tells me they will evaluate my case. I'll have a decision in a week.

At 5am, I sign off. Back to bed.

​

One hour later, 6am, the baby wakes up, as she always does. It's April 1st, the start of a new day :)

The day is fuzzy.

Lack of sleep, intense Thailand heat, and a busy baby keep me distracted. While my subconsious brain seems to be doing a lot of thinking. It also seems to be telling me to get back to the computer.

In the meantime, I'm not on the frontpage of HN anymore, but I'm still #5 on Show HN.

In the evening. At 8pm, when the baby is asleep, I find the problem easily. A clean solution isn't available. I decide to cut the Geocoding API altogether, and upload all the airport coordinates manually to my database. Not fun, but the cleanest solution for now.

Saturday evening 10pm, the site goes live again. And continues to get traffic. It's still #6 on Show HN.

I go to bed, and hope the baby will sleep throughout the night.

I also hope I didn't make a mistake. These Google Cloud emails and Citi text messages shortened my life expectancy.

What I learned and what I'm frustrated about

I learned not to build YOLO api connections when my credit card is attached!

Then again, a thread about huge bills is pinned on the Google Cloud Reddit page.

Clearly I'm not the only one.

And so I'll blame Google as well:

• For having a billing dashboard with 5 times the same balance information, displayed in a different way, but what is worse, each of them displaying different balances!
• On top, the information that one should really have, the link to the API usage, is totally absent.
• Similarly, the email about cloud costs is completely devoid of information. Why are they able to bill so fast, but can't send a warning about massive API usage?
• And finally, why aren't there easy ways to set limits to cut the usage once it crosses a certain threshold? (I found out later you can set a quota, which is in a completely different section - the Google Maps section - of the dashboard).

Overall the Console is way too bloated. And I thoroughly regret now building it on top of Google Maps, though admittedly:

• The customer service is great, it's 4am Thailand time on a Sunday morning and I'm speaking to 3 different service reps with a total waiting time of less than 2 minutes. Kudos for that.
• And two weeks after the blow-up, on Monday April 17, I receive an email that Google will refund me.

The Reddit thread mentioned there seems to be a policy around reimbursing the first time. It was the case for me. My intention is not to try twice 😉.

​

For comparison. For another project, I use the FlightAware API, which has a clear connection between calls and cost. And mentions the 10 minute delay on the graph.

Though admittedly, I haven't used it in a high stress situation (I hope I won't).

​

Postscript - 2 days later

Sunday April 2nd, two days after putting the website on Hacker News.

It's 2am. The baby starts crying.

She goes back to sleep easily.

I open my phone and I see a text message from Citi. My heart misses a beat. What happened??

It's Google, they charged me 7,520 THB (~200 USD). It's the regular billing apparently, as Saturday was April 1st, and billing happens on the first of the month.

After I shut the website down, an additional 7,520 THB was incurred. The Console interface is definitely not live. But I'm happy, as it's not an unexpected 30,000 THB again. I go back to sleep.

In the morning, 6am, the first thing I do again, is check my phone. All seems good.

I'll go through the habit of nervously checking my phone a couple more times that day.

I'm afraid additional bills might come, given the delay on the information in the console. Or because I might have made a mistake when fixing the code.

It causes an anxiety I will carry for a couple of days (and nights)

Postscript - 3 days later

I mentioned before Google didn't send any email about increased traffic on my API usage.

3 days later, on Monday April 3rd, 8am, I receive an email from Google that my usage is significantly up in the last 24 hours.

It started peaking more than 70 hours ago ...

Something wrong with the automated emails?

​

​

​

​

​
​Unsubscribe · Preferences​