Mint redefined personal finances for me: automatic categorization, syncing with my bank, real time alerts, mobile and web apps. In the aftermath of the Equifax breach, I felt scared about having all my financial history in one place with millions of other people. It was time to take control of my financial data.

A Little Background on Me

My income was beginning to feel stretched thin as my family grew from 2 to 4. Childcare costs climbed to 50% of our income, and we couldn't save money. I feel strongly that Facebook, Google, and Amazon are undermining civil society, democracy and capitalism, so I doubted there was huge earning potential for me somewhere else. Oh, and, we wanted to remodel our house...Suddenly, I really needed a budget.

Spoiler alert: I still need a budget. Lots of blog posts on these topics explain magic solutions, but this post is an odyssey and confessional through all of my failed approaches to date. It's 50% therapy, 30% competitve landscape, and maybe 20% technical problems and solutions.

Requirements

Our ultimate goal was to formulate and follow a realistic budget to find enough money for our big expenses.

Requirements:

  • Download and classify transactions automatically
  • Help with spending:
    • Reactively -- alert about overspending
    • Proactively -- check remaining budget before buying
  • Banks, credit cards, and retirement accounts
  • iPhone, Android, MacOS
  • No 3rd parties
  • Empower my non-tech life partner

After reading the NY Times article about technology-based domestic partner abuse, I've started taking that last bullet point more seriously. Each of those stories boiled down to one partner setting up the smart home, and then post-breakup using aspects of the smart home to harrass the other person. So, if both partners are familiar and comfortable with all the important tech in their life, this abuse should mostly be mitigated. For personal finances, this means there must be a simple enough user interface accessible on both our smart phones.

Nice to haves:

  • Double-entry accounting
  • Export / import to ledger
  • Good visualization
  • Customizable categorization -- an area where I wanted more than Mint provided

Looking at Alternatives

There's loads of options out there. I only looked at open source options that let me own the data. That eliminated things like LunchMoney, YooNeedABudget, and other paid options.

ledger / hledger / etc

I spent quite a bit of time tracking my finances with these tools, and they are incredible. I felt so on top of where my money was going when I used these. I'm convinced that double-entry accounting in ledger-style is what I want personally under-the-hood for personal finances. You can teach yourself a lot of accounting just from the documentation in these projects, and I've that helpful in my professional life and in various roles in my HOA and other civic groups that handle money.

Ledger & Friends Pros:

  • Double-entry accounting
  • No 3rd parties
  • Portable data format
  • Mature technology
  • Excellent documentation
  • Active community
  • Can support any currency or commodity

Ledger & Friends Cons:

  • Can't pull data from my bank / credit cards -- manual step here
  • No real time alerts
  • Single device / terminal or desktop website UIs
  • Limited categorization functionality
  • Can't imagine my partner using this

GnuCash

If you prefer GIMP to Photoshop, you might prefer GnuCash to Quicken. I haven't used this tool extensively, but it seems like it has a community behind it, and it's the only tool I looked at that can help with tax preparation. Folks running a small business are probably more the target market for this tool, rather than personal finance folks.

GnuCash Pros

  • Private - No Third Parties
  • Double Entry Accounting
  • Active development and community
  • Strong support for Quicken exports
  • Feature rich
  • Desktop GUI (and Android too)
  • Multi-currency
  • Internationalization

GnuCash Cons

Basically the same as ledger & friends...

Firefly III & Build-a-Mint-Clone & Friends

There's a ton of tools like Firefly III. Some of them are double entry accounting, others are not. Many of them can import CSVs and Quickbook files. Some of these tools are desktop apps (not multi-device) and others have web interfaces. Fava is the closest to what I want in this category. It can be accessed via a website (so that counts as multi-device) and it's a near cousin to ledger.

None of these work for me because none of these tools can sync data from banks without using a 3rd party API. Also, I'm not sure I could get my partner to use any of them.

Summary of Alternatives

No alternatives had these three features at the same time:

  • Automatic download of transactions
  • No third party data sharing
  • Multi-device
  • Compelling for my partner

I felt like I had an opportunity here to solve my own problem and possibly bring a new product to the market.

Dead End 1: React & RethinkDB

Back in 2016, I started using RethinkDB to store the transactions, and React to render charts and graphs about my transactions. I figured out how to import CSVs from one of my bank, and started calculating net income. This project was a lot of fun, but didn't really solve the hardest problem at all: downloading financials automatically.

Around this time, RethinkDB Inc shutdown, and so I removed the backend and made this an entirely JS + HTML project without a server. I realized you could probably build a 100% offline client to handle this, if you were willing to make the user send you CSV files from the bank themselves.

My work and home life got very difficult, and I gave up for at least a year.

Learnings

  • Client side JavaScript could probably do a lot of stuff I hadn't consider before
  • Babel/webpack/frontend toolchains are the absolute worst for prototyping if you don't regularly work on the frontend full time (hello vue.js!)
  • Downloading transactions should be the first problem I solve because everything else is probably easy

Dead End 2: Curl

There were awful fires in California in 2017, and then again in 2018 and our windows didn't seal at all. I worried about exposing our children to Beijing-level air-quality year after year, and so we wanted to replace our windows. The costs were staggering to me, and given our childcare costs, there was no way we could afford it without serious cost-cutting. I decided to try making a budgeting tool again.

Remembering that downloading transactions was the hardest thing, I start thinking about this first. I used the network tab in my browser to see how to login to the bank and download a CSV of transactions. Then I tried copying those commands as curl calls. I figured that the cookies would have all the details necessary to prove I was logged in and that the bank's server software would see curl commands copied from my browser as equivalent to the real thing.

I was completely wrong. After many nights and weekends, I discovered the credit union used an ancient ASP server framework that serialized and hashed all of the session state, including initial session ids, inside various form elements on the page. Every time I clicked on a link or button inside my browser on their website, the client side code would take the form state from the previous request scattered across form elements, and then produce a new hashed response code in a mangled field name that varied, and return that to the server. Without this complex dance of tightly coupled client and server code, it was impossible to move through the UI. I could only find this information after a trip down the wayback machine when trying to identify the server framework identifying some cryptic fields being based to backend. So, the chances of me being able to find working documentation for the exact version of this ancient proprietary framework used by my local credit union were slim.

Learnings

  • My bank's website might be slightly harder to hack than I first thought -- go bank!
  • I would need to use a real browser to login and download transactions

Dead End 3: Selenium

I spent the next two or three weeks of 15min sessions writing up some Selenium scripts in Python to login to the bank and download transactions. Having solved this and run the code at least twice, I felt sure I had nailed this.

Port to Rust

I love Rust a lot and wanted to spend my spare time writing software in Rust. So I started porting the Python to Rust. Unfortunately, Rust libraries for Selenium aren't nearly as feature rich and mature (or ergonomic) as those in Python. The maintainers I chatted with were super helpful and willing to accept PRs and feedback, and I was able to get something sort of working.

Challenges with Selenium

Having now run these scripts over a tethered connection 5-10 times in a row per session (there were many sessions) and over wifi in various locations, I started hitting random failures. Sometimes, it was just timeouts for finding a link on the page were too low, but other times the bank would put another page in the login flow to announce a promotion (student loans or something) and my script would break 1 in 4 times. Sometimes at night or when tethered or running the script a lot, the bank would require an SMS second factor authentication. Score one for bank security noticing I was a bit odd for logging in over and over!

State Machines

After listening to a podcast about the Automat library, I realized that I could solve these problems with a state machine. I basically need to make a graph of state transitions, and iteratively complete a state transition (enter my username and click "login") then detect the next state. (I'm not quite sure if this technically fits into the definition of a "state machine", but its close.)

I realized I could solve the 2fa problem by connecting an IFTTT SMS-trigger on my phone to a webhook to my webapp, and then parse out the text message for the one time passcode. This could be incorporated into the state machine.

It kinda worked! Sometimes, as long as I didn't do the SMS thing. But I had several accounts with other financial institutions I wanted to scrape. I worried about how I could possibly find the mental health and time to write another version of this for every single bank out there.

Code Quality Issues

My code was really hard to work with. I hacked it together when I was at my most tired, often forcing myself to code in the middle of the night or during my 20 minute bus ride to work. It was mostly a 1000 line function that panicked without real error messages when anything went wrong. Code was repeated all over the place and there were no tests.

To solve these problems, I spent many hours architecting the program. I tried to follow a Domain Driven Design, with various abstractions to represent Banks, Transactions, Categories and Budgets. There were RFC-style plans for everything I wanted to accomplish. But the more planning I did, the more dread and resentment I felt at trying to tackle such a large project without getting enough sleep or having anyone to help me, and trying to do so with a difficult programming language and small ecosystem.

I was just super unhappy with life due to work and family and neighbor stuff going on, and I gave up for many months.

Learnings

  • I needed to be more surgical in both planning and writing code
  • YAGNI -- I spent too much time planning out a fully baked program when I hadn't proven out the basic approach was scalable to multiple financial instutions
  • I've been at this for years already (no one on Hacker News seems to spend years doing anything), so I need to take time to relax and sleep and only do this when it's fun
  • As in UX testing, Selenium can be really fragile, especially if screenshot support is missing from your library

Current (Not-Yet-Working) Approach: Email

At this point of burnout and frustration, I read several posts on Hacker News (which I can no longer find) about using transaction alert emails to track personal finances. Each of these blog posts focused on the simplicity of this approach (especially compared to Selenium), but no one wrote about the security implications.

Instead of entrusting your bank password to questionable browser hijacking code running unsupervised inside your bank account, you configure your bank to send emails about every transaction. Basically every bank supports this, so you don't have to wait for banks to have proper APIs.

Assuming you setup a separate email address for this purpose with a different password than your bank account, then there's nothing for a hacker to exploit. All they can do is read about your finances, not steal all your money! Also, Selenium can't fuck up your bank account on accident either. Once I thought through the security, I completely gave up on Selenium.

IMAP -> MySQL -> Grafana

Learning from my recent burnout on this project, I decided to write the minimum amount of code as fast as possible. The best advice for this I've found is to reuse as much open source software as possible. Basically, I would write a script in Rust to check emails then parse them for transactions details, then write that information to MySQL. I could visualize that information using Grafana. No web server, no frontend code, very little math.

I got this working in under a month of nights and weekends. But as soon as I tried to visualize the data, it was clear that the data was wrong. Really wrong. I was pretty discouraged. I felt like Grafana wasn't going to be a great option for building an easy interface for my partner to use. Given the data wasn't right, I decided to remove Grafana and focus on cleaning the data.

Data Cleanup

Digging into the data, I noticed that any transfers between accounts were being double counted. So, that's sort of a tricky problem. I still haven't really solved that one yet. Instead, I decided to focus on one single account, and tackle cross-account issues once I was more confident in the basic data.

Looking into one account at a time, I started finding lots of problems:

  • Lots of transactions were missing
  • There were extra transactions, with totally wrong amounts
  • All the dates were wrong
  • Some transactions were doubled

This past month is the point were I'm finally starting to find really interesting details about how banks send emails. So, this is sign I'm possibly doing something right?

Missing Transactions

My first pass at parsing emails involved very detailed regexes. These were often only right for some of the messages. Instead, I've taken to just writing regexes that use a lot of blah blah .* blah blah in them. This seems to work much better.

Also, multi-part emails kicked my ass for a long time. Like, there would be HTML and text versions of a message inside the same email. But also some messages from the same bank would straight up text or just HTML. So, I was finally able to unify the code to handle that by making a list of parts of an email (the list might only have one element), then iterating over the list and stripping any HTML tags out of HTML emails. Then I would take the first element from the list for parsing with my regexes. This seems to work great!

Extra Totally Wrong Transactions

Turns out these emails were actually messages about declined transactions. These email notifications would make for great real time notifications once I take the time to parse those emails and build them into the application somehow. But also it seems valid that declined transactions are not listed in my transaction history on the website. This is area of functionality I hadn't anticipated and that I probably couldn't build using Selenium.

All the Dates were Wrong

In banking, there's generally two dates associated with any transaction: the posted date and the effective date. The posted date is when the bank first learned about this transaction. The other date is called the "effective" date, and this is when the bank formally "clears" the transaction. If the bank doesn't clear the transaction immediately, it could be considered "pending" for up to 2 or 3 business days, and it probably didn't affect your balance until it clears.

All that to say, for debit card transactions on checking accounts, I get emails for the posted date so that's why the dates in MySQL always a day or two before what my bank shows. This might be a WONTFIX issue, unless I can think up a clever way to solve it later.

Doubled Transactions

Related to the last section, sometimes I get a second email that's identical to the first email, when the transaction "clears." But not always.

So, I think I've only just solved this problem in the last week. Before I insert a new row into MySQL, I construct an ID in my script. The ID is a hash of only some columns I'll insert:

  • Transaction date
  • Description
  • Amount
  • Account

It doesn't include the email date nor email id (did you know emails have uids?? I only recently learned that from the IMAP protocol). Now, when there are both posted and effective date emails for the same transaction, I get MySQL errors about duplicate IDs. Success!

Not Quite Success

So It turns out, in rare cases there really are multiple transactions on the same day with the same description and amount and account. Gah! How do you distinguish between these cases?

You can't. Not using transaction emails.

Daily Account Balances

However, banks also provide daily account balance emails, so we should at least be able to detect when the actual bank balance isn't the sum of the transactions from the previous day.

Also, tracking the balance of an account (even if its only daily) is super valuable as an independent data point for my application.

How Close Am I?

It's really hard to say.

I've been focused on creating a dead simple cli in Rust, and I'm learning interesting things about the IMAP protocol, accounting, and Rust. My data is quickly becoming more accurate, but it's still not right. I'm also learning how to work on side projects more effectively by breaking down work into multi-hour tasks, and how to prioritize the right next step. It's actually fun and I have some momentum. I hangout with friends and (rarely) rage program anymore.

But nothing comes for free. I still mostly sacrifice sleep or time with friends or that just using ledger directly so I can work on this project. Honestly, I haven't 100% proven that emails can accurately track my net worth. Once that (hopefully) pans out, there's a killer interface for personal finances and setup an LLC and marketing and yada yada to do.

But I'm still optimistic, and for now I'm going to keep chugging away because this is one way I choose to try to create meaning for myself (and there's still nothing out there that quite does what I want).