A small sample can describe itself precisely, and it can suggest a range of plausible values for the mean it came from. What it cannot do is guarantee that the range is right, or that a pattern you notice is real. This guide walks through what sample size changes, what it leaves alone, and how to run the bounded resampling experiment in the 13 Labs experiment without overclaiming.

Three questions a sample size actually answers

When you look at a set of numbers, sample size affects three things and not a fourth. It determines which summary statistics you can compute at all, how stable those statistics are from sample to sample, and how much a single extreme value can drag a mean around. It does not determine whether your sampling was representative; that is a design question, not an arithmetic one.

The NIST/SEMATECH Engineering Statistics Handbook illustrates the split nicely. In its beam-deflection case study, a set of 200 observations is summarized with a sample size, mean, median, minimum, maximum, range and standard deviation, and only then are randomness and constancy of scale examined with lag, autocorrelation and spectral plots. Summary statistics came first; the validity of the model behind them was checked separately (s1). A short list of numbers deserves the same order of operations.

What a resample of sample means shows

The experiment draws subsamples with replacement from your dataset, computes the mean of each subsample, and plots the collection of those means as a dotplot. On a default dataset of 2, 3, 3, 4, 5, 6 and 100 the mean is 123/7 ≈ 17.571 and the median is 4. Draw samples of size two and their means can vary widely. For this dataset, the theoretical bounds are 2 and 100: sampling with replacement permits either extreme to be drawn twice. The observed range depends on the seed and number of repeats. Draw subsamples of size seven — the whole dataset, drawn with replacement — and the means cluster much more tightly around the same central level. That tightening is the practical content of "more data, less wobble."

Two cautions belong here. First, the resampling distribution is an illustration computed from your values, not a measurement of the world; treat it as a picture of how the mean of these numbers behaves under repeated sampling, labeled as illustrative. Second, with replacement the same value can appear more than once in a sample. A size-seven sample can differ from the original dataset. Repeated samples can therefore produce different means even at that size; increasing sample size generally reduces the variation, but any finite run can fluctuate.

What the handbook says about resampling scale

NIST's discussion of the bootstrap plot is direct about repetition counts: to estimate the uncertainty of a median from a dataset of 50 elements, a subsample of 50 elements is generated and the median computed, and this is repeated "at least 500 times" so that there are at least 500 medians. The handbook concedes that "the number of bootstrap samples to use is somewhat arbitrary" while noting that 500 subsamples is usually sufficient (s4).

This demonstration permits 50–500 repeats. Its lower settings are below the handbook's described 500-repeat benchmark; they are intended to make finite-run variation easy to explore, not to satisfy a statistical analysis standard. If a 60-repeat run and a 400-repeat run look similar, that is expected; the shape of the cloud is what you are reading, not a precise quantity. The handbook also warns that the bootstrap "is not appropriate for all distributions and statistics" and gives the range as an example of a statistic heavily dependent on the tails (s4). This experiment deliberately plots means only, for that reason.

What small n leaves you unable to claim

  • A spread is not a significance verdict. A narrow cloud of resampled means does not license a p-value. The experiment does not compute inferential statistics, and neither should you read them in.
  • Outliers dominate. On the default dataset the median is 4 while the mean is 17.571 — the single value 100 accounts for almost all of the gap. NIST's beam-deflection analysis removed three outliers and reported "a 5 % reduction in the residual standard deviation," then called the choice between the two fits "a judgment call" (s3). Removing a point to get a cleaner picture is a decision to document, not a correction to hide.
  • A small sample does not certify randomness. The transmittance case study in the handbook has 50 points and found that the measurement system sampled too quickly, contaminating each reading with the previous one (s3). No amount of mean arithmetic would have caught that; the plot and the process did.
  • No predictions. Resampling your existing values does not tell you what the next measurement will be.

A worked walkthrough, stated as hypothetical

The following is a hypothetical illustration of the tool's mechanics, not a record of any real observation.

  1. Open /experiment/. The dataset field holds the demo values 2, 3, 3, 4, 5, 6, 100. Leave them.
  2. Set sample size to 2 and repeats to 100. Click Run. Read the seed and settings printed above the dotplot and the matching rows of the accessible results table.
  3. Without changing the seed, raise the sample size to 7. Run again. The dotplot columns visibly narrow; the center stays near the same place.
  4. Change repeats to 500, keeping everything else fixed. Compare the table's summary rows to the 100-repeat run. The difference should be modest and is a reminder of the handbook's "somewhat arbitrary" note.
  5. Use Download to save the dataset, results and settings as JSON so the exact run can be checked later.

What you can say when you are done

Safe sentences look like: "Across subsamples of size two drawn with replacement from these seven values, the subsample means ranged roughly from 2.5 to 53 in this illustrative run, with a stated seed." Sentences to avoid: "The mean is significantly different," "the true value is X," or "this predicts a mean of Y." A second, longer dataset from the same source will always be a better next step than a bigger resample of the one you already have.

Keep reading

Continue with One outlier can move an average for the mean-versus-median comparison, and Read the experiment without overclaiming for language discipline. About 13 Labs explains the project's scope.