The Math Behind a Conjoint Survey
I wrote this as the math reference for my team on a systems-analysis group project: a conjoint survey testing which features of a conflict early-warning tool practitioners actually valued — forecast horizon, geographic granularity, spillover modeling, cost. The team needed to read a Sawtooth output and trust what it meant; this is what I put together so the choice data and the part-worths it produces stop being a black box.
The thread back to the SCAD spillover work is not incidental — the hypothetical tool in this survey is built around the same spillover concept, tested on practitioners instead of on conflict events. Two different methods, same underlying question: does spillover actually matter to the people who’d use a tool built on it.
Start at the top and read straight through the first time. After that, each section below is self-contained — math is introduced with plain-English motivation before the symbols, so skip to the prose if you lose the thread and come back to the equations after.
1. What the Survey Is Actually Measuring
When a respondent picks Option A over Option B in a choice task, they are not directly telling us why. They are giving us a revealed preference — a single binary signal. The entire machinery of conjoint analysis is built to reverse-engineer the hidden values behind those choices.
The central assumption is called random utility theory (Thurstone 1927, McFadden 1974). It says that every person carries around an internal score — a utility — for any product or service they encounter, and they choose whichever option has the highest utility. The word “random” is there because we, the analysts, can’t observe everything that goes into that score. Some of it is systematic (the features of the product), and some of it is noise (mood, inattention, idiosyncratic taste).
Formally:
\[U_{ic} = V_{ic} + \varepsilon_{ic}\]
Where:
- \(U_{ic}\) = the true (unobservable) utility that respondent \(i\) assigns to concept \(c\)
- \(V_{ic}\) = the part we can model, based on the product’s attributes
- \(\varepsilon_{ic}\) = random error we cannot observe
Respondent \(i\) chooses concept \(c^*\) if and only if:
\[U_{ic^*} > U_{ic} \quad \text{for all } c \neq c^*\]
The modeling challenge: we observe the choices, we know the attributes, we need to recover \(V_{ic}\).
2. The Raw Ingredients — Attributes, Levels, and the Design Matrix
2.1 Survey Structure
| Attribute | Levels | Label |
|---|---|---|
| Forecast Horizon | 1 month, 3 months, 6 months | \(A_1\) |
| Geographic Granularity | Country, Province, District | \(A_2\) |
| Spillover Modeling | None, Basic, Advanced | \(A_3\) |
| Alert Confirmation Rate | Low, Moderate, High | \(A_4\) |
| Transparency | Limited, Summary, Full | \(A_5\) |
| Implementation Cost | $0–25K, $25–100K, $100–500K | \(A_6\) |
6 attributes × 3 levels each = 18 total parameters to estimate (before any constraints).
2.2 Dummy Coding vs. Effects Coding
To plug attributes into a regression, each level gets converted to numbers. There are two standard approaches.
Dummy coding (easiest to interpret): for an attribute with 3 levels, create 2 binary variables. The omitted level is the reference category.
Example — Forecast Horizon: \(x_1 = 1\) if “3 months”, \(x_2 = 1\) if “6 months”, reference “1 month” is both = 0.
Effects coding (preferred in conjoint): same structure, but the omitted level gets coded as \(-1\) instead of \(0\). This forces utilities to sum to zero within each attribute, which makes attribute importance calculations cleaner.
\[\sum_{l=1}^{L_a} w_{al} = 0 \quad \text{for each attribute } a\]
Where \(w_{al}\) is the part-worth (utility) for level \(l\) of attribute \(a\), and \(L_a\) is the number of levels in attribute \(a\). With effects coding, the intercept becomes the grand mean utility rather than the utility of the reference level. Sawtooth uses effects coding by default; the formulas below assume it.
2.3 The Design Matrix X
For a given concept \(c\) shown to respondent \(i\), assemble a row vector \(\mathbf{x}_{ic}\) of coded attribute levels. With 6 attributes × 2 dummy variables each, this is a row of length \(S = 12\).
The full design matrix \(\mathbf{X}\) across all respondents and tasks is:
\[\mathbf{X} \in \mathbb{R}^{(N \cdot T \cdot C) \times S}\]
Where \(N\) = number of respondents, \(T\) = tasks per respondent, \(C\) = concepts per task, \(S\) = number of coded parameters. For this survey: \(N \approx 400\), \(T = 8\), \(C = 3\), \(S = 12\), so the design matrix runs roughly \(9{,}600\) rows by 12 columns.
3. From Choices to Utilities — The Core Additive Model
The part we can model, \(V_{ic}\), is assumed to be a weighted sum of the attribute-level values present in concept \(c\):
\[V_{ic} = \mathbf{x}_{ic} \cdot \mathbf{w}_i = \sum_{a=1}^{A} \sum_{l=1}^{L_a} x_{icl} \cdot w_{il}\]
Where \(\mathbf{w}_i\) is the vector of part-worth utilities for respondent \(i\) — what we’re solving for — and \(x_{icl}\) is 1 if concept \(c\) has level \(l\) of attribute \(a\), else \(-1\) (effects coding) or 0 (dummy).
Concrete example. Suppose Respondent 42 has these part-worths (Section 4 covers where they come from):
| Level | Part-worth \(w\) |
|---|---|
| Forecast: 1 month | −0.8 |
| Forecast: 3 months | +0.3 |
| Forecast: 6 months | +0.5 |
| Geographic: Country | −0.6 |
| Geographic: Province | +0.2 |
| Geographic: District | +0.4 |
| Spillover: None | −1.1 |
| Spillover: Basic | +0.3 |
| Spillover: Advanced | +0.8 |
| Alert Rate: Low | +0.9 |
| Alert Rate: Moderate | +0.1 |
| Alert Rate: High | −1.0 |
| Transparency: Limited | −0.7 |
| Transparency: Summary | +0.3 |
| Transparency: Full | +0.4 |
| Cost: $0–25K | +1.2 |
| Cost: $25–100K | +0.0 |
| Cost: $100–500K | −1.2 |
Option A in Task 1 is: 3-month horizon, Province, Basic spillover, Moderate alert rate, Summary transparency, Moderate cost.
\[V_{A} = 0.3 + 0.2 + 0.3 + 0.1 + 0.3 + 0.0 = +1.2\]
Option B is: 6-month horizon, Country, Advanced spillover, Low alert rate, Limited transparency, High cost.
\[V_{B} = 0.5 + (-0.6) + 0.8 + 0.9 + (-0.7) + (-1.2) = -0.3\]
Option C is: 1-month horizon, District, No spillover, High alert rate, Full transparency, Low cost.
\[V_{C} = (-0.8) + 0.4 + (-1.1) + (-1.0) + 0.4 + 1.2 = -0.9\]
None option has utility \(V_{\text{None}}\), typically estimated as a separate constant, often anchored at 0. Respondent 42 would choose Option A — it has the highest \(V\).
4. Estimating the Model — MNL Logit
4.1 Why Not OLS?
The outcome is a categorical choice, not a continuous number — you can’t run ordinary least squares on “chose A, chose C, chose None.” You need a model built for categorical outcomes.
4.2 The Multinomial Logit (MNL) Model
Assuming the random errors \(\varepsilon_{ic}\) follow an independent Gumbel (Type I Extreme Value) distribution, the probability of respondent \(i\) choosing concept \(c\) from a task with \(C\) options works out to:
\[P(c \mid \mathbf{X}_i, \mathbf{w}) = \frac{e^{V_{ic}}}{\sum_{c'=1}^{C} e^{V_{ic'}}}\]
This is the softmax function. The probability of choosing any concept is proportional to the exponential of its utility, relative to the sum of exponentials across all options.
Properties worth knowing:
- Probabilities always sum to 1 across the choice set: \(\sum_{c} P_c = 1\)
- If \(V_{ic} \gg V_{ic'}\) for all \(c' \neq c\), then \(P_c \rightarrow 1\)
- If all utilities are equal, probabilities are equal: \(P_c = 1/C\)
- Including the None option: just add \(e^{V_{\text{None}}}\) to the denominator
Numerical example, continuing from Section 3:
\[P(A) = \frac{e^{1.2}}{e^{1.2} + e^{-0.3} + e^{-0.9} + e^{0}} = \frac{3.32}{3.32 + 0.74 + 0.41 + 1.00} = \frac{3.32}{5.47} \approx 0.607\]
\[P(B) = \frac{0.74}{5.47} \approx 0.135, \quad P(C) = \frac{0.41}{5.47} \approx 0.075, \quad P(\text{None}) = \frac{1.00}{5.47} \approx 0.183\]
About 61% predicted probability on Option A — consistent with the deterministic choice of A.
4.3 Maximum Likelihood Estimation
The part-worths \(\mathbf{w}\) are estimated by finding the values that make the observed choices as probable as possible. For \(N\) respondents each completing \(T\) tasks:
\[\mathcal{L}(\mathbf{w}) = \prod_{i=1}^{N} \prod_{t=1}^{T} P(c_{it}^* \mid \mathbf{X}_{it}, \mathbf{w})\]
Where \(c_{it}^*\) is the concept respondent \(i\) actually chose in task \(t\). Taking the log turns the product into a sum, which is easier to optimize:
\[\ell(\mathbf{w}) = \sum_{i=1}^{N} \sum_{t=1}^{T} \log P(c_{it}^* \mid \mathbf{X}_{it}, \mathbf{w}) = \sum_{i=1}^{N} \sum_{t=1}^{T} \left[ V_{ic_{it}^*} - \log \sum_{c'} e^{V_{ic't}} \right]\]
Maximize \(\ell(\mathbf{w})\) by iterative numerical methods (Newton-Raphson or gradient descent). There is no closed-form solution. The result is a single vector \(\hat{\mathbf{w}}\) of part-worths — one score per attribute-level combination — that best explains the aggregate pattern of choices across all respondents.
4.4 The IIA Problem
The MNL model carries a built-in assumption called Independence of Irrelevant Alternatives (IIA): if you remove one option from the choice set, the remaining options keep their relative choice probabilities.
\[\frac{P(c)}{P(c')} = \frac{e^{V_c}}{e^{V_{c'}}}\]
doesn’t change when other options are added or removed. Suppose a “High cost, Advanced spillover” option is added to this survey. MNL assumes it steals share proportionally from every other option. In reality it probably only competes with other high-cost options. IIA is violated whenever options are close substitutes for only some alternatives, which is almost always true in practice — and is the core motivation for Hierarchical Bayes.
5. Measuring Attribute Importance
Once we have part-worths, we can quantify how much each attribute matters relative to the others. The importance of attribute \(a\) is its range — the spread between its best and worst levels:
\[\text{Range}_a = \max_l(w_{al}) - \min_l(w_{al})\]
Expressed as a percentage of total importance across all attributes:
\[\text{AI}_a = \frac{\text{Range}_a}{\sum_{a'=1}^{A} \text{Range}_{a'}} \times 100\%\]
Using the part-worths from Section 3:
| Attribute | Max | Min | Range |
|---|---|---|---|
| Forecast Horizon | +0.5 | −0.8 | 1.3 |
| Geographic Granularity | +0.4 | −0.6 | 1.0 |
| Spillover Modeling | +0.8 | −1.1 | 1.9 |
| Alert Confirmation Rate | +0.9 | −1.0 | 1.9 |
| Transparency | +0.4 | −0.7 | 1.1 |
| Implementation Cost | +1.2 | −1.2 | 2.4 |
Total range = 9.6. \(\text{AI}_{\text{Cost}} = 2.4 / 9.6 = 25\%\), \(\text{AI}_{\text{Spillover}} = 1.9 / 9.6 = 19.8\%\), \(\text{AI}_{\text{Horizon}} = 1.3 / 9.6 = 13.5\%\).
Implementation Cost is the dominant driver, followed closely by Spillover Modeling and Alert Confirmation Rate. An attribute importance of 25% for cost against 14% for forecast horizon tells a policymaker that budget constraints dominate their implicit preferences more than how far ahead the system predicts — the primary prescription-generating output of the whole exercise.
What importance scores cannot tell you: they’re relative and design-specific. They depend entirely on the range of levels tested. Cost from $1K to $10M instead of $0–25K to $100–500K would make cost dominate even further. Importance scores do not generalize across studies with different level ranges — worth stating plainly whenever these get presented.
6. The Problem with Aggregate Models
The MNL in Section 4 produces one set of part-worths for the entire sample. Useful for population-level predictions, but it hides the heterogeneity that’s actually interesting.
Consider two hypothetical respondents:
| Respondent | Primary concern | Implicit priority |
|---|---|---|
| UN Field Officer | Staff safety, speed | Short horizon, high recall |
| National Policy Analyst | Budget justification | Low cost, high precision |
Average their utilities and you get a muddled middle that describes neither. Comparing Africa vs. Latin America practitioners, or data-driven orgs vs. judgment-driven orgs, needs individual-level utilities. The problem: each respondent only answers 8 tasks × 3 choices = 24 data points, and you cannot reliably estimate 12 parameters from 24 observations per person using classical MNL. The solution is Hierarchical Bayes.
7. Hierarchical Bayes
7.1 The Intuition
Hierarchical Bayes (HB) borrows statistical strength from the group to stabilize individual-level estimates: your preferences are yours, but you’re not so unusual that we should ignore what everyone else said. It assumes individual part-worths \(\mathbf{w}_i\) are drawn from a population distribution, and estimates both the individual utilities and the population distribution simultaneously.
7.2 The Two-Level Model
Level 1 (individual choice model):
\[P(c_{it}^* \mid \mathbf{w}_i) = \frac{e^{\mathbf{x}_{ic_{it}^*} \cdot \mathbf{w}_i}}{\sum_{c'} e^{\mathbf{x}_{ic't} \cdot \mathbf{w}_i}}\]
Each respondent has their own \(\mathbf{w}_i\), and their choices are governed by the MNL with those personal utilities.
Level 2 (population distribution):
\[\mathbf{w}_i \sim \mathcal{N}(\boldsymbol{\mu}, \boldsymbol{\Sigma})\]
Individual utilities are drawn from a multivariate normal with population mean \(\boldsymbol{\mu}\) and covariance \(\boldsymbol{\Sigma}\). The covariance captures how attributes correlate across respondents — people who care a lot about cost also tend to care a lot about transparency, say.
Bayes’ theorem connects the two levels:
\[P(\mathbf{w}_i \mid \text{choices}_i, \boldsymbol{\mu}, \boldsymbol{\Sigma}) \propto P(\text{choices}_i \mid \mathbf{w}_i) \cdot P(\mathbf{w}_i \mid \boldsymbol{\mu}, \boldsymbol{\Sigma})\]
Updated belief about \(\mathbf{w}_i\) (posterior) is proportional to how well \(\mathbf{w}_i\) predicts choices (likelihood), times how plausible \(\mathbf{w}_i\) is given the population (prior).
7.3 Shrinkage
The individual estimate is a weighted average:
\[\hat{\mathbf{w}}_i = \lambda_i \hat{\mathbf{w}}_i^{\text{data}} + (1 - \lambda_i) \boldsymbol{\mu}\]
Where \(\hat{\mathbf{w}}_i^{\text{data}}\) is what respondent \(i\)’s data alone would suggest, \(\boldsymbol{\mu}\) is the population mean, and \(\lambda_i\) is the weight given to individual data, increasing as the respondent provides more or more-consistent data. When a respondent’s data is sparse or noisy — they chose “None” every time — \(\lambda_i \rightarrow 0\) and their estimate collapses toward the population mean. When it’s rich and consistent, \(\lambda_i \rightarrow 1\) and the estimate is dominated by their own choices.
7.4 MCMC Estimation
The posterior distributions for \(\mathbf{w}_i\), \(\boldsymbol{\mu}\), and \(\boldsymbol{\Sigma}\) have no analytical solution. Sawtooth uses Markov Chain Monte Carlo (MCMC) — specifically Gibbs sampling — to draw thousands of samples from the posterior: start with initial guesses for all parameters, sample \(\mathbf{w}_i\) for each respondent given the current \(\boldsymbol{\mu}, \boldsymbol{\Sigma}\), sample \(\boldsymbol{\mu}, \boldsymbol{\Sigma}\) given the current \(\mathbf{w}_i\) values, repeat 10,000–50,000 times, discard the early “burn-in” draws, and average what’s left for posterior mean estimates.
What you get: a point estimate \(\hat{\mathbf{w}}_i\) per respondent, a credible interval for each utility — the Bayesian analog of a confidence interval — and full uncertainty quantification instead of a single number.
7.5 Why HB Over Aggregate MNL
| Feature | Aggregate MNL | Hierarchical Bayes |
|---|---|---|
| Estimates | One set for whole sample | One set per respondent |
| Heterogeneity | Ignored | Explicitly modeled |
| Segmentation | Post-hoc (cluster utilities) | Built in (\(\boldsymbol{\Sigma}\) captures correlation) |
| Small n per person | Fine (pooled) | Handles with shrinkage |
| IIA assumption | Yes, strictly | Relaxed via mixing distribution |
| Computation | Fast | Slow (hours on large datasets) |
8. Willingness to Pay
When one attribute is a continuous price or cost variable, part-worths convert into marginal willingness to pay (MWTP) — how much money respondents would give up to gain one unit of another attribute.
\[\text{MWTP}_a = -\frac{w_a}{w_{\text{cost}}}\]
The negative sign is because cost utility is typically negative. With cost’s three discrete levels ($12.5K, $62.5K, $300K midpoints) modeled as continuous, \(w_{\text{cost}} = \Delta\text{utility} / \Delta\text{dollars}\). For Advanced Spillover Modeling vs. None, with a spillover utility difference of +1.9 and a cost coefficient of \(-0.000015\) per dollar:
\[\text{MWTP} = -\frac{1.9}{-0.000015} = \$126{,}667\]
Respondents would pay up to roughly $127K extra per year for Advanced vs. No spillover modeling — a directly policy-actionable number, with one caution: MWTP only holds when cost is modeled linearly and the range tested is realistic. This survey’s cost levels span a wide range with non-linear spacing, worth acknowledging alongside the number.
9. The Market Simulator
After estimating part-worths, you build a market simulator — a tool predicting how the target population would distribute choices across any hypothetical set of configurations.
For each respondent \(i\) and tool \(j\): compute utility \(V_{ij} = \mathbf{x}_j \cdot \mathbf{w}_i\), then apply a choice rule to convert utilities to probabilities.
First Choice Rule: each respondent deterministically adopts the highest-utility option.
\[S_j = \frac{1}{N} \sum_{i=1}^{N} \mathbf{1}[V_{ij} > V_{ij'} \; \forall j' \neq j]\]
Simple, but ignores uncertainty — best when preferences are well-identified and the sample is large.
Logit Choice Rule: each respondent has a probability of choosing each option proportional to softmax utilities.
\[P_{ij} = \frac{e^{V_{ij}}}{\sum_{j'} e^{V_{ij'}}}, \quad S_j = \frac{1}{N} \sum_{i=1}^{N} P_{ij}\]
Allows “vote splitting” and is more realistic when preferences are noisy — Sawtooth’s default. Always include a None alternative in simulations, calibrated to the observed None rate in the conjoint tasks; without it the simulator assumes 100% of the market adopts one of the presented tools, which is never true.
Scenario simulation. Say the predictive model prescribes a tool configuration prioritizing spillover modeling. Test it against alternatives — recommended configuration, lower-cost alternative, status quo — and the simulator gives a predicted preference share for each: “our recommended configuration would be the preferred choice for approximately 55% of surveyed practitioners, compared to 25% for the lower-cost alternative.” That’s the direct link between the conjoint and the prescriptive objectives.
10. Holdout Validation
A holdout task is shown identically to all respondents, excluded from utility estimation, and used afterward to test predictive accuracy.
Predict the choice in the holdout task: \(\hat{c}_i^* = \arg\max_c V_{ic} = \arg\max_c (\mathbf{x}_{ic} \cdot \hat{\mathbf{w}}_i)\). Hit rate is the proportion of respondents where the prediction matches the actual holdout choice:
\[\text{Hit Rate} = \frac{1}{N} \sum_{i=1}^{N} \mathbf{1}[\hat{c}_i^* = c_i^*]\]
Random baseline with 3 options plus None is 25%; an acceptable HB model clears 50%, a good one 60–70%. Alternatively, compare predicted choice shares to observed holdout shares via mean absolute error — lower MAE means better calibration of the market simulator.
11. Connecting Back
The conjoint produces three outputs:
Descriptive — an attribute importance bar chart showing which of the 6 attributes drives choices most. Cost and spillover modeling were expected to dominate, and did — this describes practitioner priorities empirically instead of asking them to self-report.
Predictive — segmented utility scores across subgroups: do spillover preferences differ between Africa and Latin America practitioners, given the different conflict typologies in SCAD? Does transparency matter more for analytically-oriented, data-driven orgs than judgment-driven ones? These comparisons mirror the regional comparative analysis in the SCAD work directly.
Prescriptive — given the utility model, identify the configuration that maximizes aggregate predicted adoption within realistic cost bounds: “for organizations operating in Sub-Saharan Africa with limited budgets, the configuration that maximizes predicted adoption is X, with an estimated preference share of Y%.”
The linking hypothesis worth checking: the conjoint’s spillover modeling utility scores should correlate with the SCAD model’s finding that spillover context raises predicted deaths. If practitioners who value Advanced spillover modeling are also operating in regions where the spillover variable is a significant predictor, that’s convergent validity between the quantitative and survey-based analyses — two different methods independently pointing the same direction.
12. Quick Reference
| Formula | Purpose |
|---|---|
| \(U_{ic} = V_{ic} + \varepsilon_{ic}\) | Random utility decomposition |
| \(V_{ic} = \mathbf{x}_{ic} \cdot \mathbf{w}_i\) | Additive part-worth model |
| \(\sum_{l} w_{al} = 0\) | Effects coding constraint (within-attribute) |
| \(P(c) = \dfrac{e^{V_c}}{\sum_{c'} e^{V_{c'}}}\) | MNL choice probability (softmax) |
| \(\ell(\mathbf{w}) = \sum_i \sum_t \log P(c_{it}^*)\) | Log-likelihood to maximize |
| \(\text{Range}_a = \max_l(w_{al}) - \min_l(w_{al})\) | Attribute utility range |
| \(\text{AI}_a = \dfrac{\text{Range}_a}{\sum_{a'} \text{Range}_{a'}}\) | Attribute importance (%) |
| \(\mathbf{w}_i \sim \mathcal{N}(\boldsymbol{\mu}, \boldsymbol{\Sigma})\) | HB population distribution |
| \(\hat{\mathbf{w}}_i = \lambda_i \hat{\mathbf{w}}_i^{\text{data}} + (1-\lambda_i)\boldsymbol{\mu}\) | HB shrinkage estimator |
| \(P(\mathbf{w}_i \mid \text{data}) \propto P(\text{data} \mid \mathbf{w}_i) \cdot P(\mathbf{w}_i)\) | Bayes’ theorem for HB |
| \(\text{MWTP}_a = -w_a / w_{\text{cost}}\) | Willingness to pay conversion |
| \(S_j = \frac{1}{N} \sum_i P_{ij}\) | Market share simulation (LC rule) |
| \(\text{Hit Rate} = \frac{1}{N} \sum_i \mathbf{1}[\hat{c}_i^* = c_i^*]\) | Holdout validation accuracy |
Notation
| Symbol | Meaning |
|---|---|
| \(i\) | Respondent index (\(i = 1, \ldots, N\)) |
| \(t\) | Task index (\(t = 1, \ldots, T\)) |
| \(c\) | Concept/option index within a task |
| \(a\) | Attribute index (\(a = 1, \ldots, A\)) |
| \(l\) | Level index within an attribute |
| \(L_a\) | Number of levels for attribute \(a\) |
| \(\mathbf{w}_i\) | Part-worth utility vector for respondent \(i\) |
| \(w_{al}\) | Part-worth for level \(l\) of attribute \(a\) |
| \(\mathbf{x}_{ic}\) | Coded attribute vector for concept \(c\) seen by respondent \(i\) |
| \(V_{ic}\) | Systematic utility of concept \(c\) for respondent \(i\) |
| \(\varepsilon_{ic}\) | Random error (unobserved) |
| \(\boldsymbol{\mu}\) | Population mean of part-worths (HB Level 2) |
| \(\boldsymbol{\Sigma}\) | Population covariance matrix (HB Level 2) |
| \(\lambda_i\) | HB shrinkage weight for respondent \(i\) |
| \(S_j\) | Predicted market share for alternative \(j\) |
| \(\text{AI}_a\) | Attribute importance for attribute \(a\) |
| \(\text{MWTP}_a\) | Marginal willingness to pay for attribute \(a\) |
References: McFadden (1974) on random utility theory; Lenk et al. (1996) on HB conjoint; Sawtooth Software technical papers.