How to Make Three Paid ETA Services One Free

This is a story on how to not spend even a penny by using three ETA (estimated time of arrival) services instead of one. Everything is based on my personal experience working as a back-end developer at GoDee project. GoDee is a start-up project that offers booking seats on a bus online.

Prehistory

Copy link

GoDee is a public transportation app. Bus transportation by GoDee is more convenient than motorbikes common for Southeast Asia and cheaper than a taxi. The app-based system allows users to find an appropriate route, select the time, book the seat, and pay for the ride online. And one of the problems of GoDee is traffic jams that severely impact the user experience. Users get tired of waiting and get annoyed by trying to guess the bus arrival time. So, to make the commuting more convenient, it needed service to calculate the bus' approximate arrival time, aka ETA.

Developing ETA from scratch would take at least a year. So, to speed up the process, GoDee decided to implement the Google Distance Matrix API tool. Later they developed their own Pifia micro-service.

Problems

Copy link

Over time, the business grew, and the user base increased. We encountered a problem with increasing requests in the Google Distance Matrix API.

Why is this a problem?

Copy link

Because every request costs money, Google API provides 10.000 free queries per month, after which every 1.000 queries are charged $20. At that time, we had about 150,000 requests per month.

My mentor was very dissatisfied with that. And said that the system should change caching to store ETA every 30 minutes. At that time, the system sent requests to the Google API every 3 seconds to get fresh data. However, such a caching algorithm wasn't efficient, since minibusses were stuck in traffic. And so the distance only changed once every ten minutes. There was another nuance. For example, five users are asking for information about the same bus, and this is the same request. The cache solved this type of problem.

Alternative services

Copy link

The cache worked, but not for long since GoDee grew even further and faced the same problem — the number of queries has increased again.

It was decided to replace the Google API with OSRM. Basically, OSRM is a service for building a route based on ETA (this is a rough but the short description, if you need details, here is the link).

OSRM has one problem: it builds routes and calculates ETA without taking traffic into account. To solve this problem, I started looking for services that can provide information about traffic in the specified part of the city. HERE Traffic was providing the data I needed. After a little study of the documentation, I wrote a small code that gets traffic information every 30 minutes. And to upload traffic information to OSRM, I wrote a small script with the command ./osrm-contract data.osrm --segment-speed-file updates.csv (more details here).

Math time: every half of the hour, there is a request to HERE to get traffic information this are two requests per hour, that is, a day is 48 requests (24 * 2 = 48) and a month is about ≈ 1.488 (48*31 = 1.488) a year 17.520. Yes, we have these free requests from HERE for 15 years would be enough.

Preliminary tests showed that the service works perfectly, but there is a problem, HERE gives traffic information in "gibberish" and the data does not match the OSRM format. In order for the information to fit, you need to use another service HERE for geocoding + OSRM (for getting points on the map). This is approximately 450,000 requests per month. Later, OSRM was abandoned because the number of requests exceeded the free limit. We didn't give up and enabled the HERE Distance Matrix API and temporarily removed the Google Distance Matrix API. The logic HERE is simple: we send coordinates from point A to point B and get the bus arrival time.

After we installed everything on the test server and started checking, we received the first feedback from the testers. They said that ETA reads the time incorrectly. We started looking for the problem, looked at logs (we used Data dog for logs), logs, and tests showed that everything works perfectly. We decided to ask about the problem in a little more detail, and it turned out that if the car is in traffic for 15 minutes, ETA shows the same time. We decided that this is because of the cache because it stores the original time and does not update it for 30 minutes.

We started looking for the problem, at the beginning we checked the data on the web version of the HERE Distance Matrix API (which is called we go here), everything worked fine, we received the same ETA. This problem was also checked on the google map service. There was no problem. The services themselves show this ETA. We explained everything to testers and businesses, and they accepted everything.

Our team lead suggested connecting another ETA service and returning the Google API as a backup option and writing code with the logic of switching services (the switch was needed if the requests pass the free number of requests).

The code works the following way:

val = getCount() // getting the number of queries usedif getMax() <= val { // checking for the limit of free requests for the service usednewService = switchService(s) // // if the limit is reached, switch the service returnreturn newService(from, to) // giving the logic of the new service 

We found the following Mapbox service, connected it, installed it, and it worked. As a result, our ETA had:

Conclusion

Copy link

Always look for alternatives, sometimes it happens that the business doesn't want to pay the money for the service and refuses it. As a developer who has worked hard on the service, you should bring the task to real use. This article describes how we were trying to connect more services for the free use of ETA because the business did not want to pay for the service.

P.S. As a developer, I believe that if the tool is good and does its job well, then you can pay for the tool's services. Or find open source projects 😄