iris-ai/iris
1
0 Fork 0
forked from iris/iris
Code Pull Requests Activity

Describe a tree before building it, so a failing seed can be reduced

Browse Source 
The oracle grew its trees from a seed and the shrinker grew its own, with
every scenario written out on each side. So a failure the oracle found could
not be handed to the shrinker: there was no tree to pass it, only a seed, and
a seed cannot be made smaller. The shrinker could only grow its own trees and
hope to meet the same shape, which it does not -- 20,000 of its trees never
reproduced what the oracle's seed 18 shows at depth 6.

`iris::random` now answers with a `Plan`: `plan(seed, depth, &edits)` draws one
out of the random stream and `build(rsc, &plan)` makes the widgets, where
`grow` did both at once. Every draw happens in the order it always has, so a
seed still means the tree it meant -- checked by running the oracle at 1000
seeds of depth 6 before and after and getting the same three failures with the
same boxes. `Plan::smaller` reduces one, `Plan::edited` applies an `Edits` to a
tree that already exists, and `tests/scenario/` holds the fifteen cases both
rigs now run over the same trees.

A span keeps the order it holds its children in apart from the children
themselves, so detaching, attaching and reordering leave the widgets made in
the same order and two builds still line up index for index. `Tree::detached`
is gone: `Spanned::spares` is everything made for a span that it does not
hold, which is what both of those were.

`tests/cases/plan.rs` pins the three properties the rest rests on: editing a
plan is growing one with those edits, every simplification is smaller than
what it came from, and reducing ends. The second caught this change's own
defect, where dropping a side of a `Branch` duplicated another and grew the
tree by four widgets.

What it found, on its first run: `SHRINK_SEED=18 SHRINK_DEPTH=6
SHRINK_CASE=repaint-some` reduces 277 widgets to 5. A scroll inside a scroll,
the inner one owning a movable region, and only the text at the bottom marked
for redraw -- and the span lands 24px out, which is exactly the sized child's
height. `git bisect` names `95fb4f9`, where `Masked` began reporting its box
rather than its inner's size, so what the outer scroll is told its content
measures now depends on whether the inner subtree was redrawn this frame.
`tests/cases/unsettled.rs` has it written out, ignored until it is fixed.

Checked: fmt, clippy over all targets with -D warnings, the workspace tests
(79 + 11 + 15, one ignored for the defect above), and the 100-seed oracle over
all fifteen cases at depth 4. The shrinker at 400 seeds of depth 5 now fails,
which it did not before running the oracle's trees and cases: seeds 2 and 288
on region-node and 174 and 175 on repaint-some are unreduced leads.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
iris-aiandClaude Opus 5 committed 2026-09-16 15:58:31 -04:00
1 parent 4febabfd2e
commit 98d4e98a29
7 files changed
+1469 -1290
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

tests/generated.rs
+87 -547
View File
@@ -1,19 +1,21 @@
//! Random trees, checked against building the same tree cold.
//! Laying a tree out again has to land where growing it that way would.
//!
//! A frame reaches its layout by keeping most of the last one: movable regions
//! or primitive boxes rewritten, some widgets drawn again, the rest untouched.
//! The result must be the tree a cold start would have produced, so anything
//! wrongly retained shows up as a difference in somebody's box.
//! Every case is one of `scenario`'s, over the trees `iris::random` grows
//! from a seed. The fast test takes a handful of seeds and the ignored one
//! takes as many as it is asked for; both run the same cases the shrinker
//! does over the same trees, so a seed that fails here is reduced by
//!
//! `iris::random` grows the tree and `examples/random.rs` draws one. A seed is
//! the whole reproduction; `a_long_run_of_seeds_agrees` is the ignored sweep
//! for when it is worth spending the time.
//! SHRINK_SEED=<seed> SHRINK_DEPTH=<depth> SHRINK_CASE=<case> \
//! cargo test --release --test shrink -- --ignored --nocapture
//!
//! `IRIS_GENERATED_SEED`, `IRIS_GENERATED_SEEDS` and `IRIS_GENERATED_DEPTH`
//! select what the long run covers.
use std::collections::HashMap;
#[path = "scenario/mod.rs"]
mod scenario;
use iris::harness::Harness;
use iris::prelude::*;
use iris::random::{Aligns, Edits, Lens, Rng, SpanEdit, Tree, grow};
use iris::random::{Edits, plan};
use scenario::{ALL, Case, diverges, env, over_seeds};
/// How deep the generator branches. The generator widens two to four ways per
/// level, so depth is exponential in width and a deep narrow tree is not
@@ -23,562 +25,100 @@ fn depth() -> usize {
env("IRIS_GENERATED_DEPTH", 4)
}
fn env<T: std::str::FromStr>(name: &str, fallback: T) -> T {
std::env::var(name)
.ok()
.and_then(|value| value.parse().ok())
.unwrap_or(fallback)
}
/// The seeds the ordinary tests take. Eight that have never failed and one,
/// 86, that a `Scroll` fixed point once settled differently on.
const SEEDS: [u64; 9] = [1, 2, 3, 5, 8, 10, 13, 86, 98];
/// The same box, to a step of the grid per level of nesting between the two
/// ways of reaching it. A move, a repaint and a row of shares land on the
/// same number now; what is left is a box centred in a fraction of its parent
/// against the same box centred in its own pixels. A step is a thousandth of
/// a pixel, where this was a twentieth of one before any of it was on a grid.
const AGREE_STEPS: i32 = 2;
fn same_region(got: Option<PixelRegion>, want: Option<PixelRegion>) -> bool {
match (got, want) {
(Some(got), Some(want)) => {
let same = |a: Px, b: Px| (a - b).abs() <= Px::STEP.mul_int(AGREE_STEPS);
same(got.top_left.x, want.top_left.x)
&& same(got.top_left.y, want.top_left.y)
&& same(got.bot_right.x, want.bot_right.x)
&& same(got.bot_right.y, want.bot_right.y)
}
(None, None) => true,
_ => false,
}
}
fn plant(h: &mut Harness, seed: u64, edits: &Edits) -> Tree {
let (root, tree) = grow(&mut h.rsc, seed, depth(), edits);
h.state.root = Some(root);
h.frame();
tree
}
fn resize_one(h: &mut Harness, tree: &Tree, idx: usize, rng: &mut Rng) -> Lens {
let lens = [
Some(LayoutLen::px(20.0 + rng.below(180) as f32)),
Some(LayoutLen::px(20.0 + rng.below(180) as f32)),
];
h.rsc
.widgets_mut()
.set_size_rules(tree.sized[idx], lens[0], lens[1]);
lens
}
/// Changes a few of the declared sizes, and says which, so the cold tree can
/// be grown with the same ones.
fn edit(h: &mut Harness, tree: &Tree, rng: &mut Rng) -> HashMap<usize, Lens> {
let mut edits = HashMap::new();
for _ in 0..4 {
let idx = rng.below(tree.sized.len());
edits.insert(idx, resize_one(h, tree, idx, rng));
}
edits
}
/// Every declared size at once, so every reader of a size in the tree has a
/// changed descendant in the same frame and the whole dirty set has to settle
/// together.
fn edit_every(h: &mut Harness, tree: &Tree, rng: &mut Rng) -> HashMap<usize, Lens> {
(0..tree.sized.len())
.map(|idx| (idx, resize_one(h, tree, idx, rng)))
.collect()
}
/// A way of changing what a span holds. Each is a shape worth its own case:
/// taking a child out of the middle is not the same as emptying a span, and
/// adding one is not the same as adding three.
#[derive(Clone, Copy, Debug)]
enum Shuffle {
/// Every other child, so what is left is interleaved with what went.
EveryOther,
/// Everything but the first, which is the last step before empty.
AllButFirst,
/// Three more on the end at once.
AddThree,
/// The first out and three more on, so the count moves both ways.
SwapForThree,
/// One out of the middle and one on the end.
TradeOne,
}
const SHUFFLES: [Shuffle; 5] = [
Shuffle::EveryOther,
Shuffle::AllButFirst,
Shuffle::AddThree,
Shuffle::SwapForThree,
Shuffle::TradeOne,
];
impl Shuffle {
fn of(self, grown: usize) -> SpanEdit {
let all = |step: usize, from: usize| (from..grown).step_by(step).collect();
match self {
Self::EveryOther => SpanEdit {
detach: all(2, 0),
attach: 0,
},
Self::AllButFirst => SpanEdit {
detach: all(1, 1),
attach: 0,
},
Self::AddThree => SpanEdit {
detach: Vec::new(),
attach: 3,
},
Self::SwapForThree => SpanEdit {
detach: vec![0],
attach: 3,
},
Self::TradeOne => SpanEdit {
detach: vec![grown / 2],
attach: 1,
},
}
}
}
/// Applies `shuffle` to every third span, and says what it did so the cold
/// tree can be grown that way. The widgets it takes out are given back: the
/// last share of one must outlive the comparison, or its id is handed to
/// something else and the two trees stop lining up.
fn reshuffle(
h: &mut Harness,
tree: &mut Tree,
shuffle: Shuffle,
) -> (HashMap<usize, SpanEdit>, Vec<StrongWidget>) {
let mut edits = HashMap::new();
let mut detached = Vec::new();
for (idx, span) in tree.spans.iter_mut().enumerate().step_by(3) {
let span_edit = shuffle.of(span.grown);
let mut take = span_edit.detach.clone();
take.sort_unstable();
let children = &mut h.rsc[span.id].children;
// Highest first, so an index means the same child however many of
// its neighbours are going too.
for j in take.into_iter().rev() {
if j < children.len() {
detached.push(children.remove(j));
}
}
let attach = span_edit.attach.min(span.spares.len());
children.extend(span.spares.drain(..attach));
edits.insert(idx, span_edit);
}
(edits, detached)
}
/// What a widget was configured with, so a tree the generator found can be
/// written out by hand. A fuzz failure is a lead; the fast test that replaces
/// it has to be buildable from what the failure printed.
fn describe(id: WidgetId, h: &Harness) -> String {
let rules = h.rsc.widgets().size_rules(id);
let rule = |r: SizeRule| match r.exact() {
Some(len) => format!("{len}"),
None => "-".into(),
};
let align = h.rsc.widgets().alignment(id);
let side = |a: AxisAlign| {
if a == AxisAlign::NEG {
"neg".into()
} else if a == AxisAlign::CENTER {
"mid".into()
} else if a == AxisAlign::POS {
"pos".into()
} else {
format!("{:.2}", a.rel())
}
};
// A rule and an alignment are properties of whatever carries them, so
// they print with that widget rather than as widgets of their own.
let mut out = describe_widget(id, h);
if (rules.x, rules.y) != (SizeRule::Free, SizeRule::Free) {
out += &format!("[x:{},y:{}]", rule(rules.x), rule(rules.y));
}
if align != RegionAlign::default() {
out += &format!("@{},{}", side(align.x), side(align.y));
}
out
}
fn describe_widget(id: WidgetId, h: &Harness) -> String {
let label = h.rsc.widgets().label(id).to_string();
let Some(widget) = h.rsc.widgets().get_dyn(id) else {
return label;
};
let any: &dyn std::any::Any = widget;
if let Some(w) = any.downcast_ref::<Span>() {
let sign = if w.dir.sign == Sign::Neg { "-" } else { "+" };
return format!(
"Span{{dir:{:?}{sign},gap:{},n:{}}}",
w.dir.axis,
w.gap,
w.children.len()
fn check(seed: u64, depth: usize, case: Case) {
let grown = plan(seed, depth, &Edits::default());
if let Some(how) = diverges(&grown, case, seed) {
panic!(
"seed {seed} at depth {depth} differs after {}: {how}\n\
reduce it with SHRINK_SEED={seed} SHRINK_DEPTH={depth} \
SHRINK_CASE={} cargo test --release --test shrink -- --ignored --nocapture",
case.name(),
case.name(),
);
}
if let Some(w) = any.downcast_ref::<Pad>() {
let p = &w.padding;
return format!(
"Pad{{l:{},r:{},t:{},b:{}}}",
p.left, p.right, p.top, p.bottom
);
}
if let Some(w) = any.downcast_ref::<Stack>() {
return format!("Stack{{n:{}}}", w.children.len());
}
label
}
/// Every widget in one tree against the matching widget in the other. A
/// mismatch prints the widget's ancestry, marking region nodes, since where
/// two trees disagree is rarely where the cause is.
fn assert_same(seed: u64, what: &str, warm: (&Harness, &Tree), cold: (&Harness, &Tree)) {
let ((wh, wt), (ch, ct)) = (warm, cold);
assert_eq!(wt.ids.len(), ct.ids.len(), "seed {seed}: different trees");
let mut drawn = 0;
let mut wrong = 0;
for (i, (&w, &c)) in wt.ids.iter().zip(&ct.ids).enumerate() {
let (got, want) = (wh.region(&w), ch.region(&c));
drawn += usize::from(got.is_some());
// This oracle cares where rasterization lands, not whether equivalent
// arithmetic produced the same f32. Keep the tolerance to one
// twentieth of a physical pixel, while whether a widget drew remains
// exact.
if same_region(got, want) {
continue;
}
wrong += 1;
if wrong <= 3 {
let mut chain = Vec::new();
let mut at = Some(w);
while let Some(id) = at {
let active = &wh.render.active[&id];
let node = match active.move_idx == active.parent_move {
true => "",
false => "*",
};
chain.push(format!("{}{node}", describe(id, wh)));
at = active.parent;
macro_rules! case {
($name:ident, $case:expr) => {
#[test]
fn $name() {
for seed in SEEDS {
check(seed, depth(), $case);
}
println!(
"seed {seed} after {what}: widget {i}\n warm {got:?}\n cold {want:?}\n {}",
chain.join(" < ")
);
}
}
assert!(drawn > 0, "seed {seed}: nothing was drawn");
assert_eq!(wrong, 0, "seed {seed}: {wrong} widgets differ after {what}");
}
fn changed_size(seed: u64) {
let mut warm = Harness::new((900, 1200));
let grown = plant(&mut warm, seed, &Edits::default());
// Not every tree grows a declared size to change.
if grown.sized.is_empty() {
return;
}
let mut rng = Rng::new(seed ^ 0x5eed);
let sizes = edit(&mut warm, &grown, &mut rng);
warm.frame();
let mut cold = Harness::new((900, 1200));
let same = plant(
&mut cold,
seed,
&Edits {
sizes,
..Default::default()
},
);
assert_same(seed, "a size change", (&warm, &grown), (&cold, &same));
}
/// Moves one widget to a different corner of the box it is given.
fn realign_one(h: &mut Harness, tree: &Tree, idx: usize, rng: &mut Rng) -> Aligns {
let mut side = || match rng.below(4) {
0 => None,
1 => Some(AxisAlign::NEG),
2 => Some(AxisAlign::CENTER),
_ => Some(AxisAlign::POS),
};
let aligns = [side(), side()];
for (axis, align) in [Axis::X, Axis::Y].into_iter().zip(aligns) {
h.rsc
.widgets_mut()
.set_alignment(tree.aligned[idx], axis, align.unwrap_or_default());
}
aligns
}
fn changed_alignment(seed: u64) {
let mut warm = Harness::new((900, 1200));
let grown = plant(&mut warm, seed, &Edits::default());
if grown.aligned.is_empty() {
return;
}
let mut rng = Rng::new(seed ^ 0xa11);
let aligns = (0..grown.aligned.len())
.step_by(3)
.map(|idx| (idx, realign_one(&mut warm, &grown, idx, &mut rng)))
.collect();
warm.frame();
let mut cold = Harness::new((900, 1200));
let same = plant(
&mut cold,
seed,
&Edits {
aligns,
..Default::default()
},
);
assert_same(seed, "an alignment change", (&warm, &grown), (&cold, &same));
}
/// Giving a widget a movable region of its own, or taking it away, is a
/// structural change: every primitive under it changes which chain resolves
/// it. A cold tree built that way is what says the rebuild was complete.
fn changed_region_node(seed: u64) {
let mut warm = Harness::new((900, 1200));
let grown = plant(&mut warm, seed, &Edits::default());
if grown.nodes.is_empty() {
return;
}
let nodes: HashMap<usize, bool> = (0..grown.nodes.len())
.step_by(2)
.map(|idx| {
let id = grown.nodes[idx];
let was = warm.rsc.widgets().is_region_node(id);
warm.rsc.widgets_mut().set_region_node(id, !was);
(idx, !was)
})
.collect();
warm.frame();
let mut cold = Harness::new((900, 1200));
let same = plant(
&mut cold,
seed,
&Edits {
nodes,
..Default::default()
},
);
assert_same(
seed,
"a region-node change",
(&warm, &grown),
(&cold, &same),
);
}
fn reshuffled(seed: u64, shuffle: Shuffle) {
let mut warm = Harness::new((900, 1200));
let mut grown = plant(&mut warm, seed, &Edits::default());
// Some seeds grow nothing but wrappers, and a shuffle with no span to
// shuffle is not the same thing as one that had no effect. A span behind
// a branch nobody took is the same kind of nothing: it is not drawn, so
// shuffling it cannot move anything.
let shuffles = grown
.spans
.iter()
.step_by(3)
.any(|span| warm.region(&span.id.id()).is_some());
if !shuffles {
return;
}
let (spans, _held) = reshuffle(&mut warm, &mut grown, shuffle);
warm.frame();
let mut cold = Harness::new((900, 1200));
let same = plant(
&mut cold,
seed,
&Edits {
spans,
..Default::default()
},
);
let what = format!("{shuffle:?}");
assert_same(seed, &what, (&warm, &grown), (&cold, &same));
}
fn changed_every_size(seed: u64) {
let mut warm = Harness::new((900, 1200));
let grown = plant(&mut warm, seed, &Edits::default());
if grown.sized.is_empty() {
return;
}
let mut rng = Rng::new(seed ^ 0xa11);
let sizes = edit_every(&mut warm, &grown, &mut rng);
warm.frame();
let mut cold = Harness::new((900, 1200));
let same = plant(
&mut cold,
seed,
&Edits {
sizes,
..Default::default()
},
);
assert_same(seed, "every size at once", (&warm, &grown), (&cold, &same));
}
/// Marks a spread of widgets for redraw at once. Nothing changes, so no box
/// may either; what this exercises is the order a frame settles a dirty set
/// in, which the other cases reach one dependency path at a time.
fn repainted_together(seed: u64) {
let mut warm = Harness::new((900, 1200));
let grown = plant(&mut warm, seed, &Edits::default());
for &id in grown.ids.iter().step_by(5) {
warm.rsc.widgets_mut().get_dyn_mut(id);
}
assert!(
!warm.rsc.widgets().needs_redraw.is_empty(),
"seed {seed}: nothing was marked"
);
warm.frame();
let mut cold = Harness::new((900, 1200));
let same = plant(&mut cold, seed, &Edits::default());
let what = "many repaints at once";
assert_same(seed, what, (&warm, &grown), (&cold, &same));
}
fn resized(seed: u64) {
let mut warm = Harness::new((1920, 1200));
let grown = plant(&mut warm, seed, &Edits::default());
warm.resize((640, 900));
warm.frame();
let mut cold = Harness::new((640, 900));
let same = plant(&mut cold, seed, &Edits::default());
assert_same(seed, "a resize", (&warm, &grown), (&cold, &same));
}
fn resized_then_changed(seed: u64) {
let mut warm = Harness::new((1920, 1200));
let grown = plant(&mut warm, seed, &Edits::default());
if grown.sized.is_empty() {
return;
}
warm.resize((640, 900));
warm.frame();
let mut rng = Rng::new(seed ^ 0xb0a7);
let sizes = edit(&mut warm, &grown, &mut rng);
warm.frame();
let mut cold = Harness::new((640, 900));
let same = plant(
&mut cold,
seed,
&Edits {
sizes,
..Default::default()
},
);
let what = "a resize then a size change";
assert_same(seed, what, (&warm, &grown), (&cold, &same));
}
#[test]
fn a_changed_size_lands_where_growing_it_that_way_would() {
SEEDS.into_iter().for_each(changed_size);
}
#[test]
fn a_changed_alignment_lands_where_growing_it_that_way_would() {
SEEDS.into_iter().for_each(changed_alignment);
}
#[test]
fn a_toggled_region_node_lands_where_growing_it_that_way_would() {
SEEDS.into_iter().for_each(changed_region_node);
}
#[test]
fn every_size_changing_at_once_lands_where_growing_it_that_way_would() {
SEEDS.into_iter().for_each(changed_every_size);
}
#[test]
fn many_widgets_redrawing_at_once_leaves_every_box_where_it_was() {
SEEDS.into_iter().for_each(repainted_together);
}
#[test]
fn a_resize_lands_where_starting_at_that_size_would() {
SEEDS.into_iter().for_each(resized);
}
#[test]
fn a_size_change_after_a_resize_lands_the_same_way() {
SEEDS.into_iter().for_each(resized_then_changed);
}
case!(
many_widgets_redrawing_at_once_leaves_every_box_where_it_was,
Case::RepaintSome
);
case!(
everything_redrawing_at_once_leaves_every_box_where_it_was,
Case::Repaint
);
case!(
a_resize_lands_where_starting_at_that_size_would,
Case::Resize
);
case!(
a_resize_and_a_repaint_land_where_starting_that_way_would,
Case::ResizeRepaint
);
case!(
a_size_change_after_a_resize_lands_the_same_way,
Case::ResizeSize
);
case!(
a_size_change_lands_where_growing_it_that_way_would,
Case::Size
);
case!(
every_size_changing_at_once_lands_where_growing_it_that_way_would,
Case::EverySize
);
case!(
an_alignment_change_lands_where_growing_it_that_way_would,
Case::Align
);
case!(
giving_and_taking_a_movable_region_rebuilds_what_resolves_it,
Case::RegionNode
);
case!(
reordering_a_span_lands_where_growing_it_that_way_would,
Case::Reorder
);
#[test]
fn adding_and_removing_span_children_lands_where_growing_it_that_way_would() {
for shuffle in SHUFFLES {
for seed in SEEDS {
reshuffled(seed, shuffle);
for case in ALL {
if matches!(case, Case::Shuffle(_)) {
for seed in SEEDS {
check(seed, depth(), case);
}
}
}
}
/// The same property over a hundred seeds and every scenario. What it has
/// found so far was never where the trees disagreed: a text measured in a box
/// it was not going to get, and a widget re-measured in a box its own answer
/// had decided. `tests/shrink.rs` is how a seed from here becomes a tree
/// small enough to read.
#[test]
#[ignore = "a hundred seeds, rather than the nine the others check"]
#[ignore = "as many seeds as it is asked for, rather than the nine the others check"]
fn a_long_run_of_seeds_agrees() {
let seeds = std::env::var("IRIS_GENERATED_SEED")
let depth = depth();
let seeds: Vec<u64> = match std::env::var("IRIS_GENERATED_SEED")
.ok()
.and_then(|seed| seed.parse().ok())
.map(|seed| seed..=seed)
.unwrap_or_else(|| 1..=env("IRIS_GENERATED_SEEDS", 100));
over_seeds(seeds.collect(), |seed| {
changed_size(seed);
changed_every_size(seed);
repainted_together(seed);
resized(seed);
resized_then_changed(seed);
for shuffle in SHUFFLES {
reshuffled(seed, shuffle);
}
});
}
/// Every seed on its own thread's share of them. A tree is grown, laid out
/// and dropped inside one call, so seeds share nothing, and this is most of
/// the time a run takes. A thread that fails takes the scope down with it,
/// which is the same panic libtest would have seen.
///
/// One core short of all of them, so the machine this runs on stays usable.
pub fn over_seeds(seeds: Vec<u64>, run: impl Fn(u64) + Sync) {
let threads =
std::thread::available_parallelism().map_or(1, |n| n.get().saturating_sub(1).max(1));
let chunk = seeds.len().div_ceil(threads).max(1);
std::thread::scope(|scope| {
for part in seeds.chunks(chunk) {
let run = &run;
scope.spawn(move || part.iter().for_each(|&seed| run(seed)));
.and_then(|v| v.parse().ok())
{
Some(seed) => vec![seed],
None => (1..=env("IRIS_GENERATED_SEEDS", 100_u64)).collect(),
};
over_seeds(seeds, |seed| {
for case in ALL {
check(seed, depth, case);
}
});
}