Using Groups
This page expands on the Proctor quick start guide and provides guidance about using each Proctor component. You should be familiar with the test specification, code generator and loader concepts.
Setting Up Groups
- Create a test specification: org/example/proctor/ExampleGroups.json
ExampleGroups
andExampleGroupsManager
generated by the Java codegeneratorExampleGroups.js
generated by the JavaScript codegenerator- test-matrix compiled by the Proctor builder
-
Create an instance of the AbstractJsonProctorLoader, schedule it to refresh every 30 seconds, and create an instance of
ExampleGroupsManager
using the following loader:final JsonProctorLoaderFactory factory = new JsonProctorLoaderFactory(); // Loads the specification from the classpath resource factory.setSpecificationResource("classpath:/org/example/proctor/ExampleGroups.json"); // Loads the test matrix from a file factory.setFilePath("/var/local/proctor/test-matrix.json"); final AbstractJsonProctorLoader loader = factory.getLoader(); // schedule the loader to refresh every 30 seconds final ScheduledExecutorService scheduledExecutorService = Executors.newSingleThreadScheduledExecutor(); scheduledExecutorService.scheduleWithFixedDelay(loader, 0, 30, TimeUnit.SECONDS); // Create groups manager for application-code usage final ExampleGroupsManager exampleGroupsManager = new ExampleGroupsManager(loader);
How to Determine Groups
When initializing an application’s AbstractGroups class, such as ExampleGroups, a best practice is to use the generated AbstractGroupsManager API, such as ExampleGroupsManager, instead of using Proctor
directly.
public ProctorResult determineBuckets(Identifiers identifiers
[,context-variable_1 ... ,context-variable_N]) { ... }
public ProctorResult determineBuckets(HttpServletRequest request,
HttpServletResponse response,
boolean allowForceGroups,
Identifiers identifiers
[,context-variable_1 ... ,context-variable_N]) { ... }
The generated AbstractGroupsManager handles several things:
- Uses the most recently loaded test-matrix provided by the AbstractProctorLoader implementation.
- Returns a default ProctorResult if AbstractProctorLoader returns a null
Proctor
instance. - Maps application context variables to their variable names as defined in the specification context.
- Overrides groups specified via a url-parameter or cookie-value for internal testing (if
allowForceGroups
istrue
).
Where to Determine Groups
Most applications should initialize their applications in a single location in code to ensure usage consistency across groups. For web applications, this usually means placing the group initialization early in the request-response lifecycle so groups are across requests. The group determination must be after all context
variables and test identifiers can be resolved for a given request. Your application’s values for this information will impact where the group determination can happen.
Indeed uses the ACCOUNT
identifier for loggedin-account-based tests and language
and country
context parameters for locale-specific tests. Because of this, the groups determination must come after the code that identifies a user based on their cookies and determines the language for this request.
In practice, this occurs in a javax.servlet.Filter or Spring org.springframework.web.servlet.HandlerInterceptor. Other web frameworks have their own injection points to pre-process requests.
Example Spring interceptor for determining groups:
package com.indeed.example.proctor; | |
import org.springframework.web.servlet.handler.HandlerInterceptorAdapter; | |
import javax.servlet.http.HttpServletRequest; | |
import javax.servlet.http.HttpServletResponse; | |
public class ExampleGroupsInterceptor extends HandlerInterceptorAdapter { | |
private final ExampleGroupsManager manager; | |
public GroupsInterceptor(final ExampleGroupsManager manager) { | |
this.manager = manager; | |
} | |
@Override | |
public boolean preHandle(HttpServletRequest request, | |
HttpServletResponse response, | |
Object handler) throws Exception { | |
// 1. create the set of test-identifiers and their values for a given request | |
final String trackingCookie = getTrackingCookie(request); | |
final Identifiers identifiers = Identifiers.of(TestType.USER, trackingCookie); | |
// 2. application-specific context variables | |
final com.indeed.example.UserAgent useragent = UserAgent.parse(request); | |
final String country = getCountry(request); | |
final boolean loggedIn = isLoggedIn(request); | |
// 3. can groups be forced via url-parameters / cookie values. eg. Internal-Users can force groups | |
final boolean allowForceGroups = isAllowForceGroups(request); | |
ProctorResult result = ProctorResult.EMPTY; | |
try { | |
// 4. determine buckets from groups-manager | |
result = manager.determineBuckets(request, | |
response, | |
identifiers, | |
allowForceGroups, | |
country, | |
loggedIn, | |
useragent); | |
} catch (Exception ignored) { | |
// 5. handle exceptions from javax.el expressions / rules | |
result = ProctorResult.EMPTY; | |
} | |
// 5. construct groups from the result | |
final ExampleGroups groups = new ExampleGroups(result); | |
// 6. set groups as request attribute for use in other parts of your web application | |
request.setAttribute("__groups__", groups); | |
return true; | |
} | |
public String getTrackingCookie(HttpServletRequest request) { | |
// application specific logic to identify tracking cookie | |
} | |
public boolean isLoggedIn(HttpServletRequest request) { | |
// application specific logic to identify logged-in users | |
} | |
public String getCountry(HttpServletRequest request) { | |
// application specific logic to identify country | |
} | |
private boolean isAllowForceGroups(HttpServletRequest request) { | |
// application specific logic to permit the overriding of groups | |
} | |
} |
NOTE: Lines 39-41 catch all exceptions including those thrown when evaluating the javax.el
rule expressions. Indeed logs these exceptions and uses empty-groups instead of erroring during the request.
Using the Generated Code
Java
The groups generated code contains convenience methods for accessing the each test’s group and checking if a given group is active.
The ExampleGroups.json specification will generate the following ExampleGroups.java class (the method implementations have been removed for brevity):
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
public class ExampleGroups extends AbstractGroups {
public ExampleGroups(final ProctorResult proctorResult) {
super(proctorResult);
}
public enum Test {
BGCOLORTST("bgcolortst");
}
public enum Bgcolortst implements Bucket<Test> {
INACTIVE(-1, "inactive"),
ALTCOLOR1(0, "altcolor1"),
ALTCOLOR2(1, "altcolor2"),
ALTCOLOR3(2, "altcolor3"),
ALTCOLOR4(3, "altcolor4");
}
// test bucket accessor
public Bgcolortst getBgcolortst() { }
public int getBgcolortstValue(final int defaultValue) { }
public boolean isBgcolortstInactive() { }
public boolean isBgcolortstAltcolor1() { }
public boolean isBgcolortstAltcolor2() { }
public boolean isBgcolortstAltcolor3() { }
public boolean isBgcolortstAltcolor4() { }
// Payload accessors
public @Nullable String getBgcolortstPayload() { }
public @Nullable String getBgcolortstPayloadForBucket(final Bgcolortst targetBucket) { }
}
Unassigned -> Fallback
Previous releases of the proctor library allowed users to be placed in an unassigned
bucket that differed from the fallback bucket. This is no longer the case. The generated code will now consider unassigned users to be a part of the fallback bucket declared in the project specification.
Sometimes inactive != control
For many tests the fallback bucket displays the same behavior as the control group. This is typical if you want to place your users into equal-sized buckets but do not want to set the test-behavior to 50%.
For example, you’re showing an experimental design to 10% of your users. The remaining 90% will continue to see the status-quo (control). It’s common to allocate three buckets: inactive
(80%), control
(10%), test
(10%), and mark the inactive
bucket as the fallback. Users in the inactive
bucket will experience the same behavior as users in the control
. Although 90%
of users will experience the status-quo design, only 10%
will be labeled as control. This makes absolute comparisons across metrics easier because the control
and test
groups sizes are of equal size.
When inactive and control users experience the same behavior, the following code correctly displays the new feature to the test group.
if(groups.isFeatureTest()) {
// do test
} else {
// fall through for "control" and "inactive"
}
However, there are situations where control != inactive
.
For example, you might decide to integrate with google-analytics help to track your tests by setting a custom variable. Users in the control
and test
buckets should include a unique GA variable value so their traffic can be segmented in google-analytics. The inactive
users will not get a custom variable. Your application code should explicitly handle the test
and control
buckets separate from the inactive
users.
if(groups.isFeatureTest()) {
// do test
} else if(groups.isFeatureControl()) {
// do control
} else {
// fall through for "inactive"
}
In general:
- Program towards the
test
behavior, not thecontrol
behavior. - Create an
inactive
bucket rather than assigningcontrol
as the fallback bucket.
map payload returns
Using a map payload would instead return
public ExampleGroupsPayload.Bgcolortst getBgcolortstPayload() { }
Where ExampleGroupsPayload is another generated class, and to retrieve variable values make a call similar to
final String groupVarOne = exampleGroupsInstance.getBgcolortstPayload().getVariableOne();
final Double[] groupVarTwo = exampleGroupsInstance.getBgcolortstPayload().getVariableTwo();
// returns expected variable type
// VariableOne and VariableTwo are the names of your variables inside the map payload, capitalized first character
json payload returns
json payloads provide an additional method to attempt retrieving the payload as the provided class type inputted as the parameter, if the method fails to cast the json to the provided class it returns null:
public @Nullable <T> T getExampleGroupsPayload(final Class<T> payloadType);
JavaScript
After you use your generated Java code to determine buckets for each of your test groups, you can pass the allocations to your generated JavaScript.
A method in AbstractGroups
will create an array of group allocations and payload values. Pass it an array of Test
enums from Test.values()
.
public <E extends Test> List<List<Object>> getJavaScriptConfig(final E[] tests);
You can serialize this result to JSON or write it out to the page in JavaScript. Then init()
your groups and use them as needed.
var proctor = require('com.indeed.example.groups');
proctor.init(values);
...
if (proctor.getGroups().isFeatureTest()) {
//do test
}else {
//fall through for "control" or "inactive"
}
How to Record Group Assignment
The AbstractGroups
interface provides a toString()
method that contains a comma-delimited list of active groups. Only tests with non-negative bucket value will be logged. This follows the convention that the inactive
fallback bucket has a value of -1
and doesn’t contain test behavior. The string representation for each group is [test_name][bucket_value]
.
Consider the following specification:
{
"tests" : {
// Test different single-page implementations (none, html5 history, hash-bang )
"historytst": { "buckets" : {
"inactive": -1, "nohistory":0, "html5":1, "hashbang":2
}, "fallbackValue" : -1 },
// test different signup flows
"signuptst": { "buckets" : {
"inactive": -1, "singlepage":0, "accordian":1, "multipage":2, "tabs":3
}, "fallbackValue" : -1 }
}
}
historytst | signuptst | Groups.toString() |
---|---|---|
inactive | inactive | ”” |
nohistory | inactive | “historytst0” |
html5 | tabs | “historytst1,signuptst3” |
Your application must decide how to log these groups and analyze the outcomes. Indeed logs all the groups on every requests and makes each test available in all analyses. In at least one situation, this has helped us identify and explain unexpected user behavior on pages not directly impacted by a test group’s behavior.
Extending Groups
Feel free to extend the generated Groups classes to provide application-specific functionality. Indeed logs additional information about the request as part of the groups string. For example, if a user is logged in, we’d append loggedin
to the group string. This simplifies downstream analysis by making these synthetic groups (labels) available to all of our tools that support group-based analysis.
You can add information to the groups string by overriding appendTestGroups
method and isEmpty
methods.
/**
* Return a value indicating if the groups are empty
* and should be represented by an empty string
*/
public boolean isEmpty() { ... }
/**
* Appends each group to the StringBuilder using the separator to delimit
* group names. the separator should be appended for each group added
* to the string builder
*/
public void appendTestGroups(final StringBuilder sb, char separator) { ... }
Forcing Groups
Groups can be forced by using a prforceGroups url parameter. Its value should be a comma-delimited list of group strings identical to the list used in the Groups.toString()
method documented above. The following example forces the altcolor2
bucket (value 1) for the bgcolortst
in the example groups:
http://www.example.com/path?prforceGroups=bgcolortst1
- When forcing groups, a test’s
eligibility rules
andallocation rules
are ignored. - Groups that are not forced will continue to be allocated.
- You can use the X-PRFORCEGROUPS request header instead of the url parameter.
- A cookie set on the fully-qualified domain, path=”/” will be set containing your forced groups. Manually delete this cookie or reset it with another prforceGroups parameter.
Viewing the Proctor State
The Proctor loader provides some state via VarExport
variables.
Proctor comes with several other servlets that can be used to display the Proctor state. Typically these pages are restricted to internal developers either at the application or Apache level.
AbstractShowTestGroupsController: Spring controller that provides three routes:
URL | Page |
---|---|
/showGroups |
Displays the groups for the current request. |
/showRandomGroups |
Displays a condensed version of the current test matrix . |
/showTestMatrix |
Displays a JSON test matrix containing only the tests in the application’s specification. |
// Routes the controller to /proctor
// Valid URLS: /proctor/showGroups, /proctor/showRandomGroups, /proctor/showTestMatrix
@RequestMapping(value = "/proctor", method = RequestMethod.GET)
public ExampleShowTestGroupsController extends AbstractShowTestGroupsController {
@Autowired
public ExampleShowTestGroupsController(final AbstractProctorLoader loader) {
super(loader);
}
boolean isAccessAllowed(final HttpServletRequest request) {
// Application restriction on who can see these internal pages
return true;
}
}
ViewProctorSpecificationServlet: Servlet used to display the application’s specification
web.xml:
<servlet>
<servlet-name>ViewProctorSpecificationServlet</servlet-name>
<servlet-class>com.indeed.proctor.consumer.ViewProctorSpecificationServlet</servlet-class>
<init-param>
<param-name>proctorSpecPath</param-name>
<param-value>classpath:/org/example/proctor/ExampleGroups.json</param-value>
</init-param>
</servlet>
<servlet-mapping>
<servlet-name>ViewProctorSpecificationServlet</servlet-name>
<url-pattern>/proctor/specification</url-pattern>
</servlet-mapping>