Blog posts from Projects (#projects)
15,000 emails. 8 months. All delivered to you from the only one roothouse.
It was June 2021 and we were trying to send emails for those joining HISHOT 2021, an online community seminar held by the Computer Science Studient Association of BINUS University (HINTI BINUS). If we decide to use Gmail, well, one day Google could ban us if we send more and more automated emails in the long-term. Good thing that the web hosting service behind himti.or.id also offers a SMTP email service, which we can use instead to send to them on behalf of noreply@himti.or.id. The web hosting provider rate-limits the inbuilt SMTP service to the maximum of 200 emails per hour. But just to be safe, we're just going to use a half of them. That means we have to schedule the emails to be sent at a constant rate. Sure, we can use a MySQL database to store the pending emails, but Reinhart doesn't want to run and cron PHP scripts inside the remote server. At the end of the day, he decided to write a Python script and run them on his own PC. But one question remains. Is there a better way than just letting that big laptop to run scripts 24/7? Do we have a more energy-efficient way to do this? Apparently before Reinhart got his first Raspberry Pi 3 as a birthday gift and later bought an Orange Pi Zero, and they're both unused. So we decided to try the smaller house, and oh boy, we made it root. Hey, you may already read about Shift's log about sending emails before. But sure, everyone's working to get the job done. Here we fetch all draft messages to the roothouse befor asking her to deliver one-by-one. And spoiler alert, the roothouse is getting bigger! Version 0: A greedy Python script. We first iterated on creating a simple Python script which does these things: Get a list of draft emails, straight from a remote MySQL database For each emails, send them via SMTP Simple, right? The original script really worked well. However, another problem came when the script needs to fetch all of the pending emails. Or in other words, if there are 1,000 email messages waiting to be sent, the script horribly fetches ALL of them before being able to be sent. Well, that's too much and we should fetch them little by little. Not to mention there could be some messages which we need to prioritize for and to let Shift recover again when the script crashes. Version 1: A slightly efficient script. Then we decided to send emails by priority levels by upgrading the database and our SQL query. And as expected, this version worked even better especially when queuing low-priority emails (e.g. attendance receipts and e-certificates). We also tried to fetch at most 50 pending emails per batch, before sending and re-checking them, so the overall delivery time can be significantly improved. Additionally, we started to accept and parse email messages written in Markdown so yeah, we can finally make your next emails from HIMTI less boring. HIMTI's Unified Registration system also used Markdown for creating event announcements, and we're very happy to integrate that into our emails. This version worked well until we had to think about delivering e-certificates for HISHOT 2021. No, not by sending a OneDrive or Google Drive link to the list of e-certificates, but attaching them directly into the message. So here comes Version 2. Version 2: Because e-certificates broke our script. Sending e-certificates are much slower than regular, plaintext/HTML/Markdown emails for one reason: they're huge. There are two main strategies for this, whether the script is instructed to fetch the file locally and add them to the email list, or by including the entire attached message into the database. We ended up choosing the latter one because that means we can also send anything beyond images and e-certificates. So as usual, we upgraded the database and script, but then the hosting provider blocked our websites for storing too much data inside the database. That's quite a bummer, though, that we have to regularly clear sent emails from the database to be able to schedule new emails for more people. Version 3: Concurrency. As this great email delivery solution for HISHOT was adopted to more HIMTI events, including TECHNO 2021, COMPUTERUN 2.0, and HIMTI ELECTION 2021, we realized that we have to upgrade our infrastructure to be able to send more emails, for more events and people! We also realized that not every message have to be sent over the himti.or.id SMTP server. For example, internal message for event committees when a new participant have registered and paid for the event. That's why we decided to use Shiftine's private email address to send these committee messages. And more than that, we start to use GitHub Actions to help us check emails more regularly, being able to sleep() when there are no more emails to send. We can also prioritize and strategize the delivery of different emails from separate events, like sending HIMTI ELECTION receipts during the day and COMPUTERUN 2.0 reminders during the night. Some personal reflections. Yeah, that was great! I was able to share my "can't hibernate, i'm overpowered" spirit to help people in the real world. And as I now help to maintain BINUS Today's list of articles as part of my chore, seeing those sent emails feels like a nostalgia for me. Oh, right, you can check out our final code at https://github.com/alterine0101/emailer.py/.
POLRI, meet Gojek.
And you might be asking? How was POLRI before the era of POLRI Super App? The Indonesian Police Force consists of many, many autonomous regions and units, which we can divide by 2 factors: what they do and serve as, as well as where do they operate. Take their headquarters in South Jakarta for example, where they have dedicated units for headquarter management (Yanma), public relations (Humas), and of course, their IT department (TIK POLRI). TIK POLRI is what we currently work for to coordinate the digitalization of POLRI services in our POLRI Super App. However, that central headquarter isn't the only headquarter POLRI have in the Republic of Indonesia, we have many, many regional headquarters ranging from POLDA (Provincial), POLRES (City or Regency-based), and POLSEK (District-based) services. In many cases, each POLDA and POLRES may have individual apps and digital services, which again are often made through this traditional practice of digitalization bureaucrazy. The result? Many local software houses, or should I say, tender winners, duplicated their own apps for use between POLDA and between POLRES all across the entire POLRI. Like this software company who mostly puts internal Police Force apps into their portfolio. And they seem still proud of it. In POLRI, I've heard that they already have at least 200, or even 400 (according to some rumors) individual apps and digital services running on the entire country. If each app is essentially a CRUD and at least have a dedicated Sign Up and Log In (SULI) features, then how much time is wasted to develop SULI features hundreds of times? These are the three main problems currently faced by, not just many, but any Indonesian government instances. These issues will definitely continue to exist in the future. However, there was an app, made by the minds of the Indonesian people, who finally teaches the government on how to do this "digitalization in the Industry 4.0 era", in the right way. Gojek, the local super app that (also) breaks government conventions. And here comes the classic, seemingly unrelated tale of Gojek. Originally founded in 2010, with official app first released in 2015, Gojek has become a trendsetter of its own. Its "you can also deliver stuff and order food right from our motorcycle driver network" thing have caused GrabTaxi (now Grab) and even Shopee, Traveloka, and AirAsia to have one. Its super app ambitions, and the record for being the first unicorn startup in Indonesia, have encouraged and made every local startup to be unicorn as well 🦄 Oh right, one of the Gojek's motto here is "Build startups within a startup". And one of the most epic moments, its big mobile app redesign in 2018 have finally made everyone wanted to be their own Gojek. I'm not kidding. Like, this is what Grab used to be back then. But since Gojek announced the new app design, Grab suddenly ended up like this. When DANA escaped from its home from BlackBerry Messenger with the launch of their dedicated app in 2019, this is entirely looks like the same Gojek design from 2018, despite blue. When state-owned banks and financial services agree to merge all their digital wallet apps into LinkAja in 2019, it certainly looks like Gojek, despite red. And, OVO, who was looking like this: ...ended up being this instead. So hear me out, EVERYONE. NOW. WANTS. URGES. TO. BE. GOJEK. And as private companies are trying to become more and more like Gojek, the Indonesian government finally start to realize that something is really, really wrong with their own apps. To list some issues: Government institutions are still stuck under "one app is only good for one CRUD service" mindset. Gojek, however, almost already had 20 services contained in one single app. So this isn't just all about building simple CRUD projects, right? Gojek, despite having substantial number of services, never got very bad app ratings in App Store and Google Play. At least not reaching below 2.5 stars. But many government apps still fall under the same landfill. Gojek build things mostly at their own expertise. By the way, have you read Gojek's official blog posts?No, not about that tasty GoFood and GoPay promos, but how they tackle and battle against their old codebase, aptly named Stan Marsh?I personally recommend reading Gojek Design/Tech/Engineering blog posts, they're great at explaining and experimenting at things. And finally, Gojek influences everyone to change. From the online drivers movement, rise in food delivery, new opportunities for cloud kitchen, up to that 2018 UI that makes it so original, to the point that other local apps are now start to look the same. Gojek have collectively taught us how to transform the Indonesia's digital economy and society, magnitudes of "better" than their own government. And so, the government shall learn and change how they approach their long-dreamed digital transformation, with all the right fundamentals and steps.
Indonesian government apps: Why do they suck?
There are many, many reasons, varying from human resources, management, UI/UX design, testing, operations, maintenance and more. But the biggest problem of all is the government itself! And here are six main things which I've summarized from hundreds of today's government app issues. Reason 1: Rapid outsourcing instead of growing and fostering the development team. Take that 1-starred M-Paspor for example. If you asked, "where are the developers behind this app so I can blame them for things?" The only answer I can say is well, they may be gone after the app's finished. Many Indonesian government bodies have outsourced developers from software houses, often offering affordable prices. And unfortunately, many of them are only focused on releasing these apps quickly and economically, but not willing to maintain them in the long term. Because of course, once the contracts are over, they're no longer obligated to maintain the app. It's all back to the government who decided whether to start another contract with them, or find another software house to continue their project. Reason 2: "Digitalization is optional" for some, just like Depok City. I have to agree with Mr. Yudianto's "cherry-on-top" comments regarding these apps. Some regional governments just made these apps feel great just to fulfill promises from their leaders' campaigns during the last election. And even worse, in case that these apps are down, many still believe that they can still return to pen-and-paper administration. https://twitter.com/fransallen/status/1463410005193883648?t=Q762AphooYZMmTzptmv0EA&s=19 Even the government of Depok still require traditional signatures in this era of digitalization. Reason 3: Overbranding, juxtaposing "electronification", "digital transformation", and "digitally serving the millenial generation" too much. Of course, I have a few examples here, but let's learn from the case of Sukuk Apps which you can download today from App Store and Google Play. By the way, since Google Translate can't understand what "sukuk" is, it's all about Sharia-bound state obligation investment. The app contains so much jargon which I don't understand, assuming that I never learned about English before. Like a bunch of mixed Indonesian and English terms if I set my default language to Indonesian. Do I feel very Jaksel to you? Another issue worth pointing out is the "e", as in "madness". Those e-e-ification on the bottom app bar tabs are pretty scary. Like "e-Market" which I simply assume to be yet another e-commerce like Amazon and Tokopedia. Or "e-Survey" which allows me to create online forms just like what I can do in Google Forms and Typeform? Alright, Ministry of Monetary, I know you're promoting the electronification of your service. But this is not what should be done. Like, on that homepage, "Underlying ..."? What message would you like to deliver with that truncated text? Reason 4: Bureaucrazy and their conventional app development culture. It is important to note their bureaucrazy, not just bureaucracy, of planning the digitalization of their own services. Again, assuming that they thought they can still safely return to pen-and-paper administration, every Indonesian government apps and digital services are always made by third party developers, who are just lucky enough to win official tenders from central and local governments. Their motivation? Build apps ASAP so we can digitalize ASAP! Well, I know, some of you might complain, mentioning the efforts of Jakarta and West Java provincial governments today, where they finally recruit dedicated teams for making apps. That's the latter part of the story, though, as even today we are still seeing this toxic culture still ongoing even in one of the largest law enforcement and public safety government organization in Indonesia. But well, for the rest of the governments, here's what conventional digitalization commonly look like: The government never excels in tech. But they do in marketing and election campaigns.That's why it's right, Mr. Yudianto, "Quality Assurance" and "UI/UX Designers" (almost) have no home at their annual job offerings. Because they don't understand the importance of them. So each year, the government estimates the funds first under their annual revenue and expenses allocation plan (APBD/APBN). The government then opens several projects up for tender. Software houses apply as candidates. The government may or may not know how to select a good candidate. So they either picked up randomly, or look at each portfolio, or choose because there are some recommendation from their personnels, aka. "orang dalam", between the tender winner and the government. The government creates and gives contracts to the software house. And as many software houses do, they quickly agree. The government may have little or no understanding on how their outsourced project managers are doing well or worse. Especially because of (1), the government never excels in tech. That is why many project managers "tried all their best", meeting everything what the government do as written in contract, despite the latent lack of programmer's soft and hard skills. Project managers, often crushed with deadlines and frustration with the government itself, forcing every member to speed up their work. That's why, no wonder these programmers (and often designers) start to ignore quality for the sake of "rapid digitalization", as I can summarize. "As long as I can still get money from it, why not just stick with PHP?" Or CRUD? or just not innovating at all while repeating their average skills just for money? https://twitter.com/pr0ph0z/status/1424030988594675713?s=20&t=zdmVQDE_PFVD0OISaT3AkA https://twitter.com/cahyadi_es/status/1424036633553166339?s=20&t=zdmVQDE_PFVD0OISaT3AkA Once the project either meets the finish line or the deadline, the software house will definitely consider that the app is "done", despite all the issues and the crunch their workers have done. The government then spends lots of marketing energy. Be it another press conference, another news article mentioning "digital transformation", or whatever the government could enforce because they are the government. Months after release, users start to complain things to the government related to their apps. Government's response: "Thank you for your response, we will try to improve our app" despite the contract is no longer in place. When netizens declared war against the (performance and quality of) government apps, there's another group of netizens "defending the app". This article should have been released on July 2022, right during the press release of POLRI Super App, but I decided to hold that for so long because of the rise of these tweets: https://twitter.com/itstigatiga/status/1544305595309690881?s=20&t=zdmVQDE_PFVD0OISaT3AkA https://twitter.com/Raider49703334/status/1544324994532245504?s=20&t=zdmVQDE_PFVD0OISaT3AkA https://twitter.com/18kabar_NSTR/status/1544586041369038848?s=20&t=zdmVQDE_PFVD0OISaT3AkA And of course, because of this as well: Wait, I'm not done yet! The government ended up choosing one of the three possible options: Renegotiate with the tender winner to extend their contract, Or hold another tender and back to Step 3, Or hold the improvement project entirely due to financial issues, so they may continue at the next year(s) and return to Step 1. Fortunately, after the initial app release in July, POLRI decided to choose Option 1. At the time of this writing our team is still improving on things, including the depreciation of earlier PolisiKu and Digital Korlantas for the sake of POLRI Super App. Reason 5: The fight between tender winners. Tender winners also tend to compete with each other, too. And you guessed it, for winning more projects and money. Even during the first cases of COVID-19 entering Indonesia, governments and tender winners decided to declare a great war. No, not the great war of fighting the pandemic. But the great war of fighting to be the best, winning COVID-19 superapp in Indonesia. This is, no joke. There are competing tender winners working at about projects to build individual COVID-19 information, contact tracing, and public control apps in Indonesia. Long story short (from my own Twitter thread), there are at least 6 projects from unrelated teams: PeduliLindungi from the Telkom Indonesia group, Bersatu Lawan COVID-19 (BLC-19), in which many of their veterans are joining into POLRI Super App, 10 Rumah Aman from the Ministry of Communication and Informatics (KOMINFO), Jakarta Tanggap Corona and JAKI from the Jakarta Smart City (JSC) team, Pusat Informasi dan Koordinasi COVID-19 Jawa Barat (PIKOBAR) from Jabar Digital Service, aka. the JSC of the West Java Province, Heck, even the Surabaya City Government created an Android app ("Surabaya Lawan COVID", not released to Google Play Store) which is shortly discontinued. Did you understand what I mean? Indonesia's Regional Autonomy laws fragment the effectiveness of these apps. Some apps are only good for their region, only, while the first three apps seems to compete for unknown reason. And I've contributed to the BLC-19 project as well, I have to admit that, I "hacked" PeduliLindungi for the sake of BLC-19's plans for creating an integrated dashboard for governments. That includes dismantling the APK to extract all possible endpoints from the PeduliLindungi's servers, and because I've noticed that an earlier iOS version of the app contains an accessible network debugger - which I can finally get their API keys! Wait, didn't you get the joke? I've hacked the Indonesian government for the sake of the Indonesian government! Fast forward to today, and this competition still exists even after the POLRI Super App's launch. Digital Korlantas (an earlier Police Force app for traffic and driving license management) are still being marketed to the public today, even though that we're (yes, on behalf of the greater TIK POLRI team) planning to deprecate the app in favor of POLRI Super App. https://twitter.com/khunxwhite/status/1587815218796953601 Reason 6: And lastly, their long-standing "one app is only good for one CRUD service" principle. This principle, if I'm not mistaken, drives back to the days of Symbian and Java feature phones. They are limited in capacity and capabilities, so there's no such thing as "superapps" in their limited app ecosystem. The Indonesian government, or that tender winners, tend to bring this principle forward into the era of smartphones. Their minds weren't enlightened with the ways of superapps, or at least, "modular app ecosystems" like LINE with all their web-based mini-apps (before they finally grew up into a dedicated super app). This is why the Semarang government decided to do that. Wait, what again? This list of 40+ apps which even my friends at Semarang didn't knew they existed. And this is exactly what they did! There's an app to book tickets for events in Semarang (Semartix), there's another two to provide feedbacks to the city government (E-Pokir and Lapor Hendi), there's also one to let residents and visitors to join the Semarang Great Sale craze, and there's one for citizen administration (Dukcapil / Si D'nOK) and so on. Even though that most of these apps are no longer available in the Google Play Store. But today, do you even bother to free up more phone space just so you can install all of these apps to get "the best Semarang City experience" in your whole life? Yet I believe if apps like Gojek and Traveloka didn't exist today, the quality of Indonesian government apps will still remain the same.
Building a sustainable app ecosystem for the Indonesian Police Force (POLRI).
We all have to admit that the Indonesian government is infamous for having poor quality apps and websites. Or at least, if we're discussing about this 5 years ago. User experience (UX) designers will be mad when reporting taxes on our electronic tax filing website, https://efiling.pajak.go.id. https://twitter.com/arieare/status/436649611927638016?s=21&t=cIY0FKj5It0l4zKLdxFODA https://twitter.com/reinhart1010/status/1434219937657024515?s=21&t=cIY0FKj5It0l4zKLdxFODA And of course, we're all familiar with 3-star, or even 1-star, Indonesian government apps for no longer being maintained anymore. Ratings for M-Paspor by Ministry of Law and Human Rights (Kemenkumham), Republic of Indonesia Also many citizens of Semarang City (not the Regency) who are still unaware of these 46 apps released by their government? And finally, many of these government apps ended up being Android-exclusive because the developers behind them can't even afford a fresh new Mac and a $99/year Apple Developer Account. On Tuesday, July 5th, 2022, the public screamed for witnessing a launch of another government app. "Come on, we've had enough poor quality apps at Wkwkland. And here's another one to end up in the same 3-star landfill?" There is a gap here. Our government's digital products are still considered by them as the cherry-on-top that makes their chaotic service look beautiful. Has the government taken seriously the role of the digital world in governance? This is not just about the central government, but also every department and local (province, regency, city) government. After all, they really like to release their own applications. Prabu Yudianto, "Logika Pemerintah 4.0: Bikin Aplikasi Banyak, tapi Nggak Terawat" (https://mojok.co/terminal/logika-pemerintah-4-0-bikin-aplikasi-banyak-tapi-nggak-terawat/) So what if I told you that this next government app is not just an app, or even a "superapp" by their own terms? What if I told you that this app is partly a gabutation. Yes, yet an official, high-quality gabutation of Reinhart. Also brought to you by the team behind Bersatu Lawan COVID-19 (BLC-19). Before we continue further, there are two important things that we will need to discuss. First is why many of these apps still suck, and how we overcome these issues in the best, possible way. And while the BLC-19 app itself has been brought down to the same landfill in favor of PeduliLindungi, ministries of the central government are still using our internal, integrated dashboard not just to monitor new cases, but also tracking down future possibilities of the COVID-19 case spikes. This blog post is very long. And indeed, it is now the longest blog post I've ever written in my life. That's why we decided to split this into 6 sections. Check out this site later because, well, I'm still not finished writing it yet. Page 1: Introduction. Page 2: Why do they suck? Page 3: The POLRI's current case of digital service fragmentation. And the point when everyone just wants to be [REDACTED]. Page 4: When the governments want to be [REDACTED], and how we tried to think beyond than just building a superapp. Page 5: My contributions and some trivias. Page 6: Conclusion and what's next. Of course, many of our great ideas are still covered in NDAs, so I also need to filter out these from our blog posts (unless if they are released).
