Why black-box tests could be a bad idea?

Blackbox tests are tests that are being done end-to-end. This means a UI automaton clicking on your mobile app that connects to the actual backend server – not prod, of course, but something close to staging. On Android specifically, you could use Espresso or UiAutomator to achieve those results.

What are the issues with black-box tests?

Flakiness

Blackbox tests are dependent on the server. Especially given that you don’t have a stable server instance and you test against staging, you are guaranteed to have issues and flaky tests. This means that these tests don’t give you the confidence that you need, just because sometimes the backend can be broken because of a failed push.

Stable connection

If you are using sockets, you know that the stability of the connection is something you can never take for granted. The same applies to executing tests against a socket connection. They tend to be flaky as you don’t have much control over the connection and when it is initialized. Which results again in those failed tests, that you simply re-run or ignore and that bring 0 benefit to your team.

Controlling and verifying the state

As we all know, staging DBs can get dropped or data can change. At some point, someone (like the manual QA in the team) may decide to use an account that is already used by the CI. And the mess becomes true. State is changed, screens, text and design do not match the ones expected in tests. And debugging this can take you ages just to get back on track.

Running once per day

Besides all of the above issues, black-box tests usually take a couple of hours to run, especially if you have lots of them. That means you can run them only once per day, usually nightly. Including all of the above flakiness, they don’t give you 100% confidence when it comes to the nightly execution. One failure and you are wondering – did I break something or is it because of a flaky test?

The Solution

To me, it seems that gray-box testing is the more convenient and secure way to proceed with end-to-end testing. Mocking the server by using a tool like the OkHttp MockWebServer or Wiremock can give you the stability and confidence you need. Yes, it can be a bit more complicated to setup and control the server state but once you get there, you have a better guarantee that you will catch an issue the moment you cause it.