An honest checklist before expanding an Android internal test

An Android internal test does not need to wait for a perfect product. It should not start with self-deception either.

The right question is not “is everything ready?”. The right question is:

What has already been validated in a build installed from Google Play, and what is still a known open item?

That distinction prevents two bad decisions:

• Delaying testing because of non-blocking open items.
• Inviting testers while a basic configuration error still exists.

1. Confirm the install came from Google Play

Before validating any SDK, confirm where the app came from:

adb shell dumpsys package com.sunstoneapps.dailysudoku | rg 'versionCode|versionName|installerPackageName'

Expected result:

versionCode=4
versionName=0.1.0
installerPackageName=com.android.vending

If installerPackageName is not com.android.vending, you are validating something else. Local APKs, bundletool, and sideloading are useful, but they do not replace the Google Play path.

2. Confirm observability before looking for bugs

Sentry needs to be enabled before you ask other people to test.

Look for this in the log:

[crash-reporting] initialize

Then confirm:

enabled=true
hasDsn=true
environment=production
release=com.sunstoneapps.dailysudoku@0.1.0+4
dist=4

If the app crashes during testing and Sentry is not ready, you lose the expensive data: the real error in a real environment.

3. Confirm source maps, not only events

A Sentry event without readable source maps is still half blind.

Minimum checklist:

• Release created.
• Correct dist.
• Source maps uploaded.
• Runtime announcing the same release/dist.
• A controlled or real error appears with a useful stack trace.

In this case, an old real error was useful:

RevenueCat API key is missing.

It confirmed that Sentry was receiving runtime events with stack traces and source maps. Later, the corrected build confirmed that the error stopped appearing.

4. Confirm Firebase is using the right project

Firebase can fail silently: events arrive, but in the wrong project.

Validate the gmp_app_id:

1:832961368149:android:043d6431c9c975f7fbc4dc

Then look for minimum events:

app_opened
shop_viewed
session_start

You do not need to map the whole funnel before starting an internal test. You need to know that the app installed from Play is talking to the correct project.

There is one trap worth writing down: DebugView can be the broken surface, not the app.

On the July pass for build 0.1.0+9, the Android emulator appeared in DebugView. Two physical Xiaomi-family devices did not: a Redmi Note 7 over USB and a POCO X6 over ADB Wi-Fi. That looked like a real problem until the SDK logs told a different story. Both physical devices printed Faster debug mode event logging enabled, sent app_opened, shop_viewed, and ad_query(_aq) with _dbg=1, used the expected gmp_app_id, and received 204 responses for GOOGLE_ANALYTICS uploads. GA Realtime also showed active users from the app.

That does not make DebugView useless. It means DebugView should not be the only gate on every Android skin. When a physical device stays invisible, record the gmp_app_id, app_instance_id, event names, _dbg=1, and the 204 upload result before calling the instrumentation broken. If Realtime or BigQuery confirms traffic, the release question changes from “is analytics dead?” to “is DebugView unreliable on this device family today?”.

5. Separate broken ads from missing inventory

In real AdMob traffic, no-fill can simply mean missing inventory. Do not automatically turn it into a crash diagnosis.

A useful first signal is:

ad_query(_aq)

It shows that the app queried the ads SDK. Visual validation is still needed:

• Banner does not break layout.
• Interstitial does not block the flow.
• UMP does not block app startup.
• Denying consent does not break navigation.

But ad_query(_aq) already proves that the app reached the real SDK.

6. Separate SDK configuration from commercial setup

RevenueCat is a good example.

Blocking SDK error:

RevenueCat API key is missing.

That error means the app did not even have the correct public key.

After fixing the key, the error changed to:

ConfigurationError: there are no Play Store products registered in the RevenueCat dashboard for your offerings

That second error is different. It shows the SDK is configured, but products and offerings are still missing.

It should not be hidden, but it is not the same category of problem.

7. Write status in operational language

A good checklist does not only say “pending”. It says the next concrete action.

Bad:

IAP pending.

Better:

RevenueCat SDK validated with Android key. Still need to register Play Billing products,
configure the default offering, and connect the Google Play service account for transaction
validation.

That kind of note saves time when you return to the project days later and no longer remember which part was real.

8. What was validated in this case

In build 0.1.0+4, installed from Google Play:

• versionCode=4.
• installerPackageName=com.android.vending.
• Sentry com.sunstoneapps.dailysudoku@0.1.0+4, dist 4.
• Firebase using the official project.
• Events app_opened, shop_viewed.
• Ads event ad_query(_aq).
• No RevenueCat API key is missing. error.
• Play Billing products and the default offering configured.
• Google Play service account validated in RevenueCat.
• A test one-time product purchase appearing in Play Console as Processed.
• No new Sentry purchase error after the successful test.

Known open items:

• Full visual ads/UMP inspection on a real device.
• Restore and negative purchase paths across a wider device/account matrix.

9. Validate the purchase where it actually happens

Seeing prices inside the app does not prove the purchase flow is ready. It proves that the product was loaded by the billing client.

The stronger signal comes from three places:

• Play Console shows the order under Order management.
• RevenueCat shows Valid credentials for subscriptions, in-app products, and monetization permissions.
• Sentry does not receive a new purchase error after the successful test.

For a test purchase, estimated revenue can still show BRL 0.00. For a smoke test, the important part is that the order is Processed, uses the expected product, and does not create a new observability regression.

10. When to expand the test

I would expand the internal test when:

• Cold start does not crash.
• Observability is correct.
• Minimum analytics works.
• Ads do not block the main flow.
• The shop does not promise working purchases if products do not exist yet.
• Known open items are written down.

You do not need to solve everything before inviting more testers. You need to avoid confusing “we know this is missing” with “we discovered this by breaking the tester’s session”.

11. Before moving from internal test to production

The production track adds one more question: is the app ready for people outside your tester list, or is the console merely letting you click the next button?

In our July release pass, the internal build was already installed from Play on a real Redmi Note 7. The package reported versionCode=9, versionName=0.1.0, and installerPackageName=com.android.vending. The app opened without a crash, Sentry announced com.sunstoneapps.dailysudoku@0.1.0+9, Firebase App Measurement uploaded events, UMP wrote consent mode, RevenueCat loaded real BRL offerings, and the AdMob banner returned no-fill without blocking the game.

That was enough to send the first production release for review, but not enough to pretend the work was over. We still kept the open follow-ups visible: confirm Sentry from its console and run a real purchase or restore with an authorized tester account when the store path is available again. For Firebase, the follow-up became more specific: DebugView worked on the emulator, while physical Xiaomi devices needed validation through SDK logs plus GA Realtime or BigQuery.

One Play Console detail mattered more than expected: production had no country availability. The release review showed a real error until the production track received countries and regions. After selecting 177 countries/regions, including “Rest of world”, the blocking error disappeared. The remaining warning was only the missing deobfuscation file for the App Bundle, useful for diagnostics but not a launch blocker for this build.

The lesson is simple: treat Play’s production review as another validation surface. It checks track availability, release notes, artifact state, and policy gates that a local build cannot see.

Run the checklist as a loop, not an event

Internal testing works best when it is repeated in the same order. A one-time smoke test can catch a crash, but a weekly loop catches drift: store configuration changes, tester account issues, billing metadata delays, consent behavior, and device-specific regressions.

The loop should be short enough that it actually happens. Install the current release build, launch fresh, check onboarding, start one core flow, exercise billing or ads if they are part of the release, and record the result. The value is the comparison with last week, not the length of the checklist.

What to record

A useful internal testing note should include:

• build version and version code;
• device model and Android version;
• Play account used for testing;
• install source and whether the app was clean-installed;
• billing, ads, consent, and crash reporting observations;
• screenshots or logs only when they explain a problem.

This keeps the checklist from becoming a diary. The note should help the next release owner answer whether behavior changed and where to look first.

Keep manual and automated testing honest

Automation is valuable, but it should not pretend to cover the Play Store. A browser or component test can validate app logic quickly. A real internal track install validates signing, store delivery, billing eligibility, consent SDK behavior, and device integration.

Both layers matter. Use automated tests to catch regressions before building. Use internal testing to verify that the release artifact behaves correctly in the ecosystem where users will get it.

Keep the checklist small enough to survive

The best checklist is the one the team still runs when tired. If it takes an hour, it will be skipped under pressure. Split it into a mandatory core and optional deep checks. The core should prove install, launch, one monetized or store-dependent flow, and one saved-state flow. Deep checks can rotate by week.

This makes the process sustainable. A short checklist that runs every release catches more regressions than a perfect checklist that only runs before big milestones.

That is the difference between a checklist and a ceremony: the checklist gives a decision, not a performance.

Conclusion

A good internal test is not a simulation of a perfect launch. It is a controlled environment for finding problems with enough instrumentation to act.

The honest checklist avoids turning testers into human logs. It makes clear what has been proven by the Google Play build and what still needs configuration before a larger beta.