How-to
How to Run Beta Testing in 7 Steps
- App store beta
- Flag-based beta
- Loglune
On this page
Beta testing gives a pre-release version of a product to a limited group of users outside the development team, so the team can find technical and user-experience problems before the general release. This guide runs beta testing in 7 steps: set the goal and the exit criteria, choose an internal, closed, or open beta, recruit and size the tester group, distribute the build or switch the beta on, collect feedback through a direct channel, record which variant each tester received, and end the beta.
The steps cover two setups: installed apps distributed through Apple TestFlight or Google Play testing tracks, and web services that switch a beta on for chosen users with a feature flag. Recruiting services and paid tester marketplaces are outside the scope.
What beta testing means
Beta testing means releasing a pre-release version to a chosen or self-selected group of users and using their reports to fix problems before everyone gets the release. Google’s Play Console Help page on how to set up an open, closed, or internal test describes testing an app with specific user groups, or with an open test on Google Play, to fix technical or user experience issues before the app is available on Google Play. Apple’s App Store Connect Help separates internal testers, who are App Store Connect users with access to the team’s content, from external testers, people the team invites who are not App Store Connect users.
For web services, Pete Hodgson’s article Feature Toggles (October 9, 2017) describes beta features that are available to internal users plus beta users, managed with permissioning toggles. Hodgson calls turning on new features for a set of internal or beta users a Champagne Brunch, and separates it from a canary release: a canary release exposes a feature to a randomly selected cohort of users, and a Champagne Brunch exposes it to a specific set of users.
| Stage | Who gets the change | How users are chosen | Source |
|---|---|---|---|
| Internal test | The team and trusted colleagues | Named on an email list or on the team’s App Store Connect account | Play Console Help, App Store Connect Help |
| Closed beta | Invited testers | Email lists, groups, or a flag rule on named users | Play Console Help, Hodgson |
| Open beta | Anyone who opts in | Self-selection through the store or a public link | Play Console Help, App Store Connect Help |
| Canary release | A random share of users | Random selection | Hodgson |
Before you start beta testing
Six inputs decide how the beta runs and when it ends.
- A build or feature complete enough for outside users, with its known issues written down.
- The question the beta must answer and the condition that ends it.
- A feedback channel testers reach from the product, such as an email address, a form, or a forum.
- For installed apps, a role that can manage testers: Apple lists Account Holder, Admin, or App Manager for external testers.
- For web services, a feature flag that can target named users by a stable user key.
- A plan for testers’ access when the beta ends.
How to run beta testing in 7 steps
1. Set the goal and the exit criteria
Write down what the beta must show and the condition that ends it before the first tester joins. For example, the beta must show that a new report builder handles the report types customers run every week, and it ends after 14 days without a new blocking report. The numbers in the example are illustrative; set them from the question the beta answers.
2. Choose an internal, closed, or open beta
Google’s Play Console Help recommends starting with an internal test and then expanding to a small group of closed testers. On Google Play, a new Android App Bundle on the internal test track becomes available to testers within minutes, a closed test reaches testers on email lists, Google Groups, or organizations, and an open test makes the test version visible on Google Play to anyone who joins. The same page says developers with personal accounts created after November 13, 2023, must meet specific testing requirements before their app can be available on Google Play.
TestFlight follows a similar order. Apple’s help page on external testers says a team must create an internal group before it can create an external group, and an external build is submitted to TestFlight App Review, where the first build a team submits requires a full review and later builds of the same version might not.
For a web service, the same order works with a flag rule: the internal team first, then a named set of beta users, then a wider group. Hodgson’s Champagne Brunch covers the first two stages, internal users and then internal users plus beta users.
3. Recruit and size the tester group
Size the group to the question: a crash that appears on some devices needs many device models, and a workflow question needs users who run that workflow every week. The platforms set upper limits, listed in the table. On TestFlight, a public link can require device and OS criteria and can carry a tester limit between 1 and 10,000, and Apple warns that anyone can share the link, so testers can join without a direct invitation.
| Platform and group | Who joins | Limit |
|---|---|---|
| TestFlight internal testing | App Store Connect users on the team’s account | Up to 100 internal testers |
| TestFlight external testing | People invited by email or through a public link | Up to 10,000 external testers per app |
| Google Play internal test | Testers on an email list | Up to 100 testers per app |
| Google Play closed test | Email lists, Google Groups, or organizations | Up to 200 lists of up to 2,000 users each, and up to 50 lists per track |
| Google Play open test | Anyone who opts in on Google Play | Unlimited, or a limit of at least 1,000 |
| Flag-based beta | Users whose attributes match the flag rule | Set by the rule |
The limits come from Apple’s and Google’s help pages linked above.
4. Distribute the build or switch the beta on
For an installed app, add the build to the tester group. In TestFlight, the team adds a build to a group and enters What to Test information that testers in every group with access to the build can read, and a team can submit up to six builds for TestFlight App Review within 24 hours. On Google Play, the first publication of a test can take several hours before the test link reaches testers, and each tester opts in through the link.
For a web service, deploy the beta code path switched off and add a flag rule that serves the beta value to the tester list. In Loglune, a flag’s targeting conditions are ordered and choose a value from the evaluation attributes, which are the customer, the user key, and other attributes, and a condition can check whether a user key is in the beta list. Publishing the draft rule creates an immutable version, and the Loglune SDK evaluates that version in your own process or at the edge.
5. Collect feedback through a direct channel
Google’s help page says testers cannot leave public reviews on Google Play for test versions and that feedback from test users does not affect the app’s public rating, so the team needs a direct feedback channel such as email, a website, or a message forum. Open and closed testers on Google Play can also send private feedback through Google Play.
Ask each report for the time, the build number, and what the tester saw. When the app attaches the Loglune evaluation record to the report, the report carries the version, the attribute key, and the returned value of the flag evaluation, and Loglune can import that record for a reproduction.
6. Record which variant each tester received
A beta group changes while the beta runs: testers join through a public link, accounts leave, and the rule widens. A report from the second week needs the rule that applied at the reported moment, not the current rule. For installed apps, TestFlight shows the installation date, sessions, and crashes of each tester who joined by email invitation, which ties a report to a build.
For a flag-based beta, Loglune appends every edit to the beta rule to the change history as a change event with the customer, the actor, the time, the target flag, and the values before and after. To answer a report, pick the tester’s customer account and the reported time T, or import the evaluation record from the report. Loglune rebuilds the flag state at T from the change history and delivers it in reproduction mode to a development, CI, or staging environment, where evaluating it with the tester’s user key and attributes returns the value the tester received. One delivery counts as one reproduction, and Loglune reproduces the flag evaluation, not the root cause in application code. The walkthrough on reproducing a bug that affects one customer covers the path from a report to a reproduction.
7. End the beta and remove the beta path
End the beta on the exit condition from step 1. On Google Play, pausing the track ends a test: testers stop receiving updates, and the app stays installed on their devices. On TestFlight, removing a build from a group leaves testers who use it with access until they switch builds or stop testing, and internal testers can download and test builds for 90 days.
For a flag-based beta, turn the rule on for every user when the feature ships, then delete the flag check from the code. Hodgson notes that a permissioning toggle for premium users can live for years, so decide at the end of the beta whether the flag becomes a release flag that is removed or a long-lived permission.
Editorial example (not a customer case): a flag-based beta for a report builder
The table follows one flag-based beta for a web service. Days, counts, and names are illustrative.
| When | Rule for report-builder-beta | Who gets the beta | Event |
|---|---|---|---|
| Day 1 | Created, off for every user | No one | Build with the new report builder deployed, beta path switched off |
| Day 2 | On for the internal team | 12 internal users | Internal checks |
| Day 5 | On for 40 invited users from 8 customer accounts | Internal team and invited users | Feedback form opens |
| Day 11, 2:10 p.m. | Unchanged | Same as day 5 | A tester’s export fails |
| Day 12 | One customer account removed at its request | Internal team and 35 invited users | Change event recorded |
| Day 13 | Unchanged | Same as day 12 | The tester from the removed account reports the day 11 failure |
| Day 13 | Flag state of that account at day 11, 2:10 p.m. rebuilt in staging | Production unchanged | The rebuilt state shows the beta value |
| Day 21 | On for every user | Everyone | Beta ends |
| Day 30 | Flag check deleted from the code | Everyone | Flag retired |
On day 13, the current rule shows the beta off for the tester’s account, because the account left the beta on day 12. The rebuilt flag state at day 11 shows that the tester received the beta path when the export failed, so the team debugged the beta path instead of the old report builder.
Troubleshooting beta testing
- Testers cannot find the app on Google Play: before open testing or a production release, internal and closed testers cannot find the app by search, so share the opt-in link or the Play Store URL. The opt-in link appears when the app status is Published, and apps in Draft or Pending publication status show no opt-in link.
- A closed tester gets the production version: Google Play delivers the highest version code across the tracks a user is eligible for, so give the test build a higher version code than the production build.
- An internal tester cannot join a closed or open test: Google Play makes users who opted into internal testing ineligible for closed and open tests until they opt out of the internal test.
- Strangers join a TestFlight beta: anyone can share a public link, so set a tester limit or device criteria, or deactivate the link.
- Reports arrive without a time or a version: add the time, the build number, and the flag evaluation record to the feedback form.
- A report arrives after the tester’s access changed: rebuild the flag state of the tester’s account at the reported time instead of reading the current rule.
FAQ
What does beta testing mean?
Beta testing means giving a pre-release version to a limited group of users outside the development team and fixing the problems they report before the general release. Google Play and TestFlight both separate internal testers on the team’s own account from wider groups of invited or public testers.
How many beta testers do you need?
No single number fits every beta, because the count follows the question the beta answers. The platforms set upper limits: TestFlight allows up to 10,000 external testers per app, and a Google Play open test is unlimited unless the team sets a limit of at least 1,000.
Can beta testers leave public reviews?
Beta testers on Google Play cannot leave public reviews for test versions, and their feedback does not affect the app’s public rating, according to Google’s Play Console Help. Open and closed testers can send private feedback through Google Play, and the team’s own channel collects the rest.
Is beta testing the same as a canary release?
Beta testing and a canary release pick users differently. Hodgson’s Feature Toggles article says a canary release exposes a feature to a randomly selected cohort of users, while a Champagne Brunch, the flag technique behind a beta, exposes it to a specific set of users.
Deciding when beta testing is worth running
Run beta testing when a change needs reports from real users that internal checks cannot produce, such as workflows that depend on customer data or device models the team does not own. Skip a separate beta for a change whose risk shows up in service metrics, which a canary release measures against a control. Keep the beta rule’s change history, so a report from any day of the beta can be matched to the flag state the tester’s account had at that moment.
Rebuild what one beta tester received at a past time
When a beta runs behind a feature flag, Loglune records every change to the beta rule as a change event with the actor, the time, and the values before and after. Pick the tester's customer account and a time T, and Loglune rebuilds the flag state that account had at T for your development, CI, or staging environment. The Free plan includes 5 reproductions.