irisandClaude Sonnet 5
7eaf79370e
Pressing Update on one client of a multi-client project ran every
component's build to get the one that was actually asked for -- cheap
while a project had one APK, expensive the moment it had two and one of
them was slow (an ARM cross-compile, say). /prepare and /build now take
?component= to restrict a run to one named component; absent still means
the whole project, which is what Pull & Build, the project-row Rebuild,
and every single-component project keep doing. There is deliberately no
component-scoped Rebuild -- forcing one component's build without
touching the rest happens by pressing Update on it, which now runs
/prepare scoped to that component.
Verified against a live server driving a scratch two-Apk project:
building one component leaves the other's marker untouched, an unknown
?component= answers 404 naming the project and the component, and naming
none still builds both.
That verification surfaced a second, sharper bug the scoping change had
not caused but did make newly visible: component_is_stale's "never built
at all" check still asked find_apks of the whole project root, left
behind when per-component discovery (c4e19c2) moved everywhere else to
each component's own directory. A project with two Apk components has
one's output sitting under the root-anchored patterns too, so the moment
either component had ever been built, the whole project read as
"something is built here" -- and the other, never built, silently stopped
being offered its own first build. /prepare saw a component with a
command and no output and declared it current. Fixed by scoping the same
check to the component's own directory, guarded to Apk components only:
a Server never has an APK to find under its directory by definition, and
asking would have reported every server "never built" forever, which
broke two existing tests before the guard was added. Component::dir is
now the one definition of what a component's directory is, used by the
build command, the staleness check, and discovery alike.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
1683 lines
70 KiB
Rust
1683 lines
70 KiB
Rust
//! Runs an app's optional on-demand build step and tracks it, so the phone
|
|
//! can be shown "building..." rather than silently downloading a stale APK.
|
|
//!
|
|
//! Nothing here knows what any particular build does: it is a command from
|
|
//! the config plus a two-file staleness rule (see
|
|
//! `crate::config::Component`). The build runs in the background and
|
|
//! the routes answer immediately either way, so a multi-minute build never
|
|
//! holds a request open past a phone's read timeout.
|
|
|
|
use std::collections::{HashMap, VecDeque};
|
|
use std::path::PathBuf;
|
|
use std::process::Command;
|
|
use std::sync::{Arc, Mutex};
|
|
use std::time::Instant;
|
|
|
|
use serde::Serialize;
|
|
|
|
use crate::config::Component;
|
|
|
|
/// What a build has got through, in whatever it counts in.
|
|
#[derive(Clone, Copy, Serialize)]
|
|
#[serde(rename_all = "camelCase")]
|
|
pub struct Progress {
|
|
pub done: u64,
|
|
pub total: u64,
|
|
}
|
|
|
|
/// The marker a build prints to report progress, at the start of a line.
|
|
///
|
|
/// Recognised on every build's output rather than switched on by a config
|
|
/// key: a build opts in by printing it, and one more `starts_with` per line
|
|
/// is not worth a field, an acceptance, and a manifest flag to avoid.
|
|
/// Nothing here knows what the numbers count -- Gradle tasks, files,
|
|
/// anything -- only that the second is the whole of it.
|
|
const PROGRESS_MARKER: &str = "@@progress ";
|
|
|
|
/// What is known about whether a component's build is current.
|
|
///
|
|
/// Three states rather than a boolean, because "we cannot tell" must not
|
|
/// share a value with either answer. A project this server has never
|
|
/// built, one outside git, and one with uncommitted work in its directory
|
|
/// are all *unknown* -- and an unknown drawn as `Current` would be the
|
|
/// exact failure this was added to stop.
|
|
#[derive(Clone, Copy, PartialEq, Eq, Debug, Serialize)]
|
|
#[serde(rename_all = "camelCase")]
|
|
pub enum Freshness {
|
|
/// Built from the commit its own directory is on.
|
|
Current,
|
|
/// Its directory has moved past the commit it was built from.
|
|
Behind,
|
|
/// Never built here, no checkout to compare against, or uncommitted
|
|
/// work in its directory.
|
|
Unknown,
|
|
}
|
|
|
|
/// How a finished component reports the commit it was built from.
|
|
///
|
|
/// A callback rather than a handle to the app list: `BuildState` lives
|
|
/// *inside* that list, so holding it would be a cycle, and the single fact
|
|
/// a build needs to report upward does not justify one. The route that
|
|
/// starts a build has the list and supplies this.
|
|
pub type RecordBuilt = Arc<dyn Fn(&str, String) + Send + Sync>;
|
|
|
|
/// The handles a build shares with the rest of the server, as opposed to
|
|
/// the project configuration it is built from.
|
|
///
|
|
/// Grouped because they travel together and because they are a different
|
|
/// kind of thing from the rest of the arguments: everything else describes
|
|
/// *this project*, while these two are the server's own state that a build
|
|
/// happens to touch -- a pull marking a checkout current, and an exec
|
|
/// waiting for a download.
|
|
#[derive(Clone)]
|
|
pub struct Shared {
|
|
pub remote_checks: crate::git::RemoteChecks,
|
|
pub downloads: Arc<crate::restart::Downloads>,
|
|
}
|
|
|
|
#[derive(Default)]
|
|
struct Inner {
|
|
/// What each component was last built from, by component name.
|
|
///
|
|
/// Seeded from the config at construction and updated as a run
|
|
/// records. Kept here rather than read back off `components` because
|
|
/// this state is deliberately *carried across* a config write -- see
|
|
/// [`BuildState::matches`] -- so its own copy of the components goes
|
|
/// stale the moment a build records against it.
|
|
built_from: HashMap<String, String>,
|
|
building: bool,
|
|
/// When the current run began, for the elapsed time shown while it is
|
|
/// still going.
|
|
started: Option<Instant>,
|
|
/// What the *project* is doing: fetching, pulling. Work belonging to
|
|
/// one component is in `runs` instead, because that is where the card
|
|
/// shows it -- a progress bar under the whole project could only ever
|
|
/// say that something, somewhere, was happening.
|
|
phase: Option<String>,
|
|
/// One entry per component the current run has reached, in the order
|
|
/// it reached them.
|
|
runs: Vec<ComponentRun>,
|
|
/// The failure from the last completed run, cleared when a new one
|
|
/// starts. Kept rather than logged-and-dropped because the phone is
|
|
/// where this is being driven from and usually has no access to the
|
|
/// server's log.
|
|
error: Option<String>,
|
|
/// Whether that failure was a pull with no fast-forward to make,
|
|
/// which is the one the phone can offer a way past. Beside the
|
|
/// message rather than inside it because the phone acts on it: a
|
|
/// button that appears only when git happened to phrase itself a
|
|
/// certain way is a button nobody can rely on.
|
|
unrelated_histories: bool,
|
|
/// The component whose step produced that failure.
|
|
///
|
|
/// Kept beside the message so the card can open the right log without
|
|
/// guessing: a component that failed to build wants its build log,
|
|
/// while every other component still wants the runtime one.
|
|
failed: Option<String>,
|
|
}
|
|
|
|
/// What one component's part of the build is doing, or did.
|
|
///
|
|
/// Per component rather than one set of numbers for the whole project,
|
|
/// because that is how it is read: a progress bar and the lines scrolling
|
|
/// under it belong to the thing being built, and shown under the project
|
|
/// they could only say that *something* was happening. It is also what
|
|
/// lets two components build at once without their output interleaving
|
|
/// into one stream.
|
|
struct ComponentRun {
|
|
name: String,
|
|
/// What it is doing now -- building, installing, restarting -- or
|
|
/// `None` once it has finished.
|
|
step: Option<String>,
|
|
started: Instant,
|
|
/// Filled in when the component finishes, so the card can keep showing
|
|
/// how long it took.
|
|
took_ms: Option<u64>,
|
|
/// Steps done and steps total, when the running command reports them.
|
|
progress: Option<(u64, u64)>,
|
|
/// The tail of this component's output. Bounded because this is a
|
|
/// progress display, not a log file -- `crate::logs` has the whole of
|
|
/// it, per component, on disk.
|
|
log: VecDeque<String>,
|
|
error: Option<String>,
|
|
}
|
|
|
|
pub struct BuildState {
|
|
/// What this project produces, in the order it produces it. A run walks
|
|
/// them, and the phase names on the status are the component names, so
|
|
/// the card shows which one is being worked on without any new
|
|
/// machinery.
|
|
components: Vec<Component>,
|
|
/// Which app this is, for naming the log files a build writes. The
|
|
/// only thing here that needs the key; everything else is about the
|
|
/// project on disk.
|
|
key: String,
|
|
/// Whether the Pull button is offered. Per project rather than per
|
|
/// component: there is one checkout.
|
|
git_pull: bool,
|
|
project_path: PathBuf,
|
|
/// True for the entry that *is* this server.
|
|
///
|
|
/// Only its `Server` component is delivered differently: that component
|
|
/// is this process, so restarting it means exec-ing the binary just
|
|
/// built (`crate::restart`) rather than asking the service script to
|
|
/// stop and start the code doing the asking -- which would kill the
|
|
/// walk partway through, with nobody left to report that it finished.
|
|
///
|
|
/// Passed in rather than read from the declaration because no project
|
|
/// may claim it: a file that could would be a file that can make this
|
|
/// server exec an arbitrary binary.
|
|
is_self: bool,
|
|
/// Force git's remote commands onto IPv4, for the fetch a pull starts.
|
|
/// Beside `git_pull` and treated the same way, including in
|
|
/// [`Self::matches`] -- toggling it has to reach the next pull, and
|
|
/// the state is reused across config writes unless something says not
|
|
/// to.
|
|
git_ipv4: bool,
|
|
/// Shared with the list, so a pull can record that this checkout is
|
|
/// current instead of leaving a stale "new commits" on the card.
|
|
shared: Shared,
|
|
inner: Mutex<Inner>,
|
|
}
|
|
|
|
#[derive(Serialize)]
|
|
#[serde(rename_all = "camelCase")]
|
|
pub struct BuildStatus {
|
|
pub stale: bool,
|
|
pub building: bool,
|
|
pub error: Option<String>,
|
|
/// That failure was a pull the checkout has no fast-forward for,
|
|
/// because it shares no history with its upstream. The phone offers
|
|
/// the forced pull for this and nothing else.
|
|
pub unrelated_histories: bool,
|
|
/// What the whole project is doing -- fetching, pulling -- absent
|
|
/// between runs and while the work belongs to a component instead.
|
|
pub phase: Option<String>,
|
|
/// Milliseconds since this run started.
|
|
pub elapsed_ms: u64,
|
|
/// Each component the run has reached, in the order it reached them.
|
|
/// The card draws each of these inside that component's own row.
|
|
pub components: Vec<ComponentStatus>,
|
|
}
|
|
|
|
/// One component's part of a build, as the phone sees it.
|
|
#[derive(Serialize)]
|
|
#[serde(rename_all = "camelCase")]
|
|
pub struct ComponentStatus {
|
|
pub name: String,
|
|
/// What it is doing now, absent once it has finished.
|
|
#[serde(skip_serializing_if = "Option::is_none")]
|
|
pub step: Option<String>,
|
|
/// How long it has been going, or took.
|
|
pub elapsed_ms: u64,
|
|
/// Steps done and steps total, when the running command reports them.
|
|
/// Absent for a command that says nothing, which is most of them --
|
|
/// the phone shows a bar that only spins rather than inventing a
|
|
/// number.
|
|
#[serde(skip_serializing_if = "Option::is_none")]
|
|
pub progress: Option<Progress>,
|
|
/// The last few lines this component printed, oldest first.
|
|
pub log: Vec<String>,
|
|
/// Why this component's step failed, if it did.
|
|
#[serde(skip_serializing_if = "Option::is_none")]
|
|
pub error: Option<String>,
|
|
}
|
|
|
|
/// Reads a progress report off a line of build output, in any of the
|
|
/// shapes this understands.
|
|
///
|
|
/// Anything that isn't exactly that shape is output, not a report: a build
|
|
/// that prints the marker with something unparseable after it has a bug
|
|
/// worth seeing in its log rather than being silently swallowed.
|
|
fn parse_progress(line: &str) -> Option<(u64, u64)> {
|
|
let line = line.trim();
|
|
if let Some(rest) = line.strip_prefix(PROGRESS_MARKER) {
|
|
let (done, total) = rest.trim().split_once('/')?;
|
|
return Some((done.trim().parse().ok()?, total.trim().parse().ok()?));
|
|
}
|
|
parse_cargo_progress(line)
|
|
}
|
|
|
|
/// Reads cargo's own counter: `Building [===> ] 236/237: crate(bin)`.
|
|
///
|
|
/// Cargo's, not this server's arithmetic -- it is the number of units it
|
|
/// has left to compile, which is exactly the thing a bar can honestly be
|
|
/// drawn from. `spawn` is what makes cargo emit it into a pipe at all.
|
|
///
|
|
/// Anchored on the bracketed bar so that a line merely containing `12/34`
|
|
/// is not read as progress: build output is full of numbers.
|
|
fn parse_cargo_progress(line: &str) -> Option<(u64, u64)> {
|
|
let rest = line.strip_prefix("Building ")?;
|
|
let counts = rest.strip_prefix('[')?.split_once(']')?.1.trim();
|
|
// `236/237: dev-updater(bin)` -- the crate after the colon is what it
|
|
// is working on now, which the log line above it already said.
|
|
let counts = counts.split(':').next()?;
|
|
let (done, total) = counts.split_once('/')?;
|
|
Some((done.trim().parse().ok()?, total.trim().parse().ok()?))
|
|
}
|
|
|
|
/// How much command output is kept for the phone. Enough to see what a
|
|
/// build is doing, far short of keeping a build log in memory.
|
|
const LOG_LINES: usize = 40;
|
|
|
|
/// Calls `emit` once per segment of a command's output, where a segment
|
|
/// ends at a newline **or** a carriage return, and once more for anything
|
|
/// left unterminated when the stream ends.
|
|
///
|
|
/// Both terminators, because a tool that redraws a counter in place ends
|
|
/// its line with `\r` and no newline. Cargo does exactly this:
|
|
///
|
|
/// ```text
|
|
/// Compiling dev-updater v0.1.0 (...)\n
|
|
/// Building [=====> ] 236/237: dev-updater(bin) \r Finished `release` ...\n
|
|
/// ```
|
|
///
|
|
/// `BufReader::lines()` splits on `\n` alone, so the count sat unread
|
|
/// until whatever printed the next newline. On a build with many crates
|
|
/// to compile that is the next `Compiling` line and the bar moved fine;
|
|
/// on an incremental one -- a single dirty crate, which is what a pull
|
|
/// produces -- it is the `Finished` line, arriving after the build is
|
|
/// over. So the card showed nothing to measure for exactly the builds
|
|
/// somebody watches, and it looked like a build system that could not
|
|
/// report progress rather than a report being held.
|
|
///
|
|
/// A `\r\n` pair ends a segment and then opens an empty one, which the
|
|
/// caller drops along with every other blank segment.
|
|
fn read_segments(pipe: impl std::io::Read, mut emit: impl FnMut(&str)) {
|
|
use std::io::BufRead;
|
|
|
|
let mut reader = std::io::BufReader::new(pipe);
|
|
// Held across fills, because a terminator is what ends a segment and
|
|
// a read can stop anywhere -- including inside a multi-byte
|
|
// character, which is safe here only because both terminators are
|
|
// ASCII and so cannot appear inside one.
|
|
let mut pending: Vec<u8> = Vec::new();
|
|
loop {
|
|
// Copied out so the borrow of the reader ends before `consume`.
|
|
let chunk = match reader.fill_buf() {
|
|
Ok([]) => break,
|
|
Ok(bytes) => bytes.to_vec(),
|
|
// The process's exit status is what says whether the build
|
|
// worked; a pipe that cannot be read further is just the end
|
|
// of what there is to show.
|
|
Err(_) => break,
|
|
};
|
|
reader.consume(chunk.len());
|
|
for byte in chunk {
|
|
if byte == b'\n' || byte == b'\r' {
|
|
emit(&String::from_utf8_lossy(&pending));
|
|
pending.clear();
|
|
} else {
|
|
pending.push(byte);
|
|
}
|
|
}
|
|
}
|
|
if !pending.is_empty() {
|
|
emit(&String::from_utf8_lossy(&pending));
|
|
}
|
|
}
|
|
|
|
impl BuildState {
|
|
pub fn new(
|
|
key: String,
|
|
components: Vec<Component>,
|
|
git_pull: bool,
|
|
git_ipv4: bool,
|
|
project_path: PathBuf,
|
|
shared: Shared,
|
|
is_self: bool,
|
|
) -> Arc<Self> {
|
|
let built_from = components
|
|
.iter()
|
|
.filter_map(|component| {
|
|
Some((
|
|
component.name().to_string(),
|
|
component.built_from()?.to_string(),
|
|
))
|
|
})
|
|
.collect();
|
|
Arc::new(Self {
|
|
key,
|
|
inner: Mutex::new(Inner {
|
|
built_from,
|
|
..Inner::default()
|
|
}),
|
|
components,
|
|
git_pull,
|
|
git_ipv4,
|
|
project_path,
|
|
shared,
|
|
is_self,
|
|
})
|
|
}
|
|
|
|
/// Whether this state was built from the same configuration, i.e.
|
|
/// whether it can be carried across a config reload -- see
|
|
/// `crate::registry::build_entries`.
|
|
/// Compared by *declaration*, not by full equality. Recording what a
|
|
/// build was made from writes the config, which rebuilds the entry
|
|
/// list -- and if that counted as a different configuration the state
|
|
/// would be replaced mid-walk, leaving the phone polling a fresh
|
|
/// `BuildState` that reports nothing is building. The same is true of
|
|
/// the package a download reads back.
|
|
pub fn matches(&self, components: &[Component], git_pull: bool, git_ipv4: bool) -> bool {
|
|
self.git_pull == git_pull
|
|
&& self.git_ipv4 == git_ipv4
|
|
&& self.components.len() == components.len()
|
|
&& std::iter::zip(&self.components, components)
|
|
.all(|(mine, theirs)| mine.same_declaration(theirs))
|
|
}
|
|
|
|
/// Idempotent: kicks off the build in the background if this app is
|
|
/// stale and no build is already running, a no-op otherwise. Safe for
|
|
/// the phone to call on every download, which is exactly what it does.
|
|
///
|
|
/// `component` narrows both the staleness check and the build itself
|
|
/// to one component -- what the phone asks before downloading it, so
|
|
/// that pressing Update on one client of a multi-client project
|
|
/// doesn't also pay for the other's build. `None` is every other
|
|
/// caller, which still means "the whole project".
|
|
pub fn trigger_if_needed(self: &Arc<Self>, component: Option<&str>, record: RecordBuilt) {
|
|
// Before the lock, not inside it -- see `is_stale`.
|
|
if self.is_stale(component) {
|
|
self.build_now(component, record);
|
|
}
|
|
}
|
|
|
|
/// Builds whether or not anything looks stale, for somebody who asked.
|
|
///
|
|
/// The staleness rules exist to keep a *download* from rebuilding the
|
|
/// world; they have no business overruling a person who pressed a
|
|
/// button. This is also what makes the freshness signal reachable at
|
|
/// all: the commit a build was made from is only recorded by a build
|
|
/// that runs here, so a project already current with its checkout
|
|
/// could never record one and would report unknown for ever.
|
|
///
|
|
/// Still idempotent -- while one run is going, another does nothing,
|
|
/// whichever component either names: there is one build slot per
|
|
/// project, not one per component, so a second request while the
|
|
/// first is still running is a no-op rather than a second concurrent
|
|
/// build. The phone notices by polling `/status` and re-reads once it
|
|
/// clears.
|
|
///
|
|
/// `component` restricts the run to one named component, or every one
|
|
/// with a command for `None` -- see [`Self::trigger_if_needed`] for
|
|
/// why a caller would want the former.
|
|
pub fn build_now(self: &Arc<Self>, component: Option<&str>, record: RecordBuilt) {
|
|
{
|
|
let mut inner = self.inner.lock().unwrap();
|
|
if inner.building {
|
|
return;
|
|
}
|
|
inner.building = true;
|
|
inner.error = None;
|
|
inner.failed = None;
|
|
inner.unrelated_histories = false;
|
|
inner.started = Some(Instant::now());
|
|
inner.runs.clear();
|
|
}
|
|
|
|
let this = Arc::clone(self);
|
|
let component = component.map(str::to_string);
|
|
tokio::task::spawn_blocking(move || {
|
|
this.run_build(component.as_deref(), &record);
|
|
});
|
|
}
|
|
|
|
/// Whether any component `name` selects is behind. Whole project for
|
|
/// `None`, which is what a run with nothing named means: one
|
|
/// component being stale is reason enough to walk the whole list, and
|
|
/// each component's own command is skipped if it has nothing to do.
|
|
fn is_stale(&self, name: Option<&str>) -> bool {
|
|
// Read once, here, and passed down. `component_is_stale` must not
|
|
// take this lock itself: it is reached from callers that already
|
|
// hold it, and a `std::sync::Mutex` is not reentrant, so doing so
|
|
// is a deadlock rather than a slow path.
|
|
let built_from = self.inner.lock().unwrap().built_from.clone();
|
|
self.components
|
|
.iter()
|
|
.filter(|component| name.is_none_or(|name| component.name() == name))
|
|
.any(|component| self.component_is_stale(component, &built_from))
|
|
}
|
|
|
|
/// What can be said about this component's build, for the card.
|
|
///
|
|
/// Deliberately not the same question as [`Self::component_is_stale`],
|
|
/// which decides whether to *run* a build. Uncommitted work makes the
|
|
/// comparison unreliable, so it reports unknown -- but it must not
|
|
/// trigger a rebuild: somebody editing on the build machine is
|
|
/// building it themselves, and this server's job in that state is to
|
|
/// notice the new output rather than to produce one. Rebuilding
|
|
/// uncommitted work and sending it to a phone would also be a
|
|
/// surprising thing to do with work its author has not committed.
|
|
pub fn freshness(&self, component: &Component) -> Freshness {
|
|
let built = self
|
|
.inner
|
|
.lock()
|
|
.unwrap()
|
|
.built_from
|
|
.get(component.name())
|
|
.cloned();
|
|
let Some(built) = built else {
|
|
return Freshness::Unknown;
|
|
};
|
|
if crate::git::subtree_dirty(&self.project_path, component.cwd()) != Some(false) {
|
|
return Freshness::Unknown;
|
|
}
|
|
match crate::git::subtree_head(&self.project_path, component.cwd()) {
|
|
Some(current) if current == built => Freshness::Current,
|
|
Some(_) => Freshness::Behind,
|
|
None => Freshness::Unknown,
|
|
}
|
|
}
|
|
|
|
fn component_is_stale(
|
|
&self,
|
|
component: &Component,
|
|
built_from: &HashMap<String, String>,
|
|
) -> bool {
|
|
if component.build().is_empty() {
|
|
return false;
|
|
}
|
|
// Nothing built at all is as far behind as an output gets, and it
|
|
// is checked before any rule rather than after: a `staleWhen`
|
|
// compares two files *inside* a build, so a component that has
|
|
// never been built reports not-stale under it and would leave the
|
|
// first build impossible to trigger -- which a project can be
|
|
// added before having done.
|
|
//
|
|
// Scoped to *this component's own* directory, not the project
|
|
// root: a project producing two APKs has one component's output
|
|
// sitting under the root-anchored patterns too (an inner
|
|
// `*/build/outputs/apk/*/*.apk` matches a one-level subdirectory
|
|
// regardless of which component it belongs to), which made the
|
|
// whole project read as "something is built here" the moment
|
|
// either component had ever been built -- masking that the
|
|
// *other* component, never built, had nothing to trigger it.
|
|
//
|
|
// Only asked of an `Apk`: a `Server` never has one to find under
|
|
// its own directory by definition, so the same question asked of
|
|
// it would report every server "never built" for ever, which is
|
|
// exactly the false staleness this check exists to rule out.
|
|
if matches!(component, Component::Apk { .. })
|
|
&& crate::discover::find_apks(&component.dir(&self.project_path)).is_empty()
|
|
{
|
|
return true;
|
|
}
|
|
// Behind the checkout: this component was built from a commit
|
|
// that its own directory has since moved past. The case nothing
|
|
// used to catch -- a pull that fetched but did not build, because
|
|
// the declaration was waiting to be accepted or because the walk
|
|
// failed earlier -- left an output older than the commits it
|
|
// should have come from, and the card had no reason to doubt it.
|
|
//
|
|
// Both halves must be known. No recorded commit means this server
|
|
// has never built it, and an unreadable checkout means it cannot
|
|
// tell; neither is evidence of staleness, and announcing one would
|
|
// make every project built before this existed demand a rebuild.
|
|
if let Some(built) = built_from.get(component.name())
|
|
&& crate::git::subtree_head(&self.project_path, component.cwd())
|
|
.is_some_and(|current| ¤t != built)
|
|
{
|
|
return true;
|
|
}
|
|
// Deliberately not the other way round: a matching commit does not
|
|
// prove currency, because a `staleWhen` rule compares two build
|
|
// outputs and can be behind for reasons no commit explains.
|
|
component
|
|
.stale_when()
|
|
.is_some_and(|rule| rule.is_stale(&self.project_path))
|
|
}
|
|
|
|
/// Fetches, fast-forwards if there is anything to take, and builds --
|
|
/// the Pull button, which acts on the build machine rather than the
|
|
/// phone. Idempotent in the same way as [`Self::trigger_if_needed`]:
|
|
/// while one is running, another does nothing.
|
|
///
|
|
/// Building happens when the pull actually moved the branch, or when
|
|
/// the configured staleness rule says the output is behind anyway; a
|
|
/// pull that finds nothing new therefore costs nothing.
|
|
///
|
|
/// `may_build` answers "may the project's command run?" -- false when
|
|
/// it is asking for a build step nobody has accepted (see
|
|
/// `AppEntry::pending_declaration`), in which case this pulls and
|
|
/// stops. The decision belongs to the caller because it is about the
|
|
/// project's config file, which is not this module's business -- here
|
|
/// it is simply "don't run the command".
|
|
///
|
|
/// It is a closure rather than a bool because of *when* it has to be
|
|
/// asked: **after the fast-forward, not before it.** The declaration
|
|
/// lives in the checkout, so the pull is the one thing that changes
|
|
/// the answer, and a bool computed at the call site is necessarily the
|
|
/// answer for the commit being replaced. Passing one meant a pull
|
|
/// whose new commits changed the declaration built anyway -- the card
|
|
/// went off and updated the app instead of stopping to ask, which is
|
|
/// the single case this gate exists for.
|
|
/// `force` abandons this checkout's own history in favour of the
|
|
/// upstream's, and is only ever pressed after a pull has already
|
|
/// reported that the two are unrelated (`git::PullError`). It is a
|
|
/// parameter rather than something decided here because it is a
|
|
/// person's answer to that report, not a state of the repository.
|
|
pub fn pull_and_build(
|
|
self: &Arc<Self>,
|
|
force: bool,
|
|
may_build: impl Fn() -> bool + Send + 'static,
|
|
record: RecordBuilt,
|
|
) {
|
|
{
|
|
let mut inner = self.inner.lock().unwrap();
|
|
if inner.building {
|
|
return;
|
|
}
|
|
inner.building = true;
|
|
inner.error = None;
|
|
inner.failed = None;
|
|
inner.unrelated_histories = false;
|
|
inner.started = Some(Instant::now());
|
|
inner.runs.clear();
|
|
}
|
|
|
|
let this = Arc::clone(self);
|
|
tokio::task::spawn_blocking(move || {
|
|
let outcome = this.pull(force);
|
|
match outcome {
|
|
Err(error) => {
|
|
tracing::error!("pull failed: {}", error.message);
|
|
this.fail_pull(error);
|
|
}
|
|
Ok(pulled) => {
|
|
// Nothing configured to build, or nothing allowed to:
|
|
// the pull was the whole job, and reporting success is
|
|
// all that is left.
|
|
// `may_build()` is called here, with the pulled
|
|
// declaration on disk, for the reason in the doc
|
|
// comment above.
|
|
if !may_build() || !this.has_command() || !(pulled || this.is_stale(None)) {
|
|
this.finish(None, None);
|
|
} else {
|
|
this.run_build(None, &record);
|
|
}
|
|
}
|
|
}
|
|
});
|
|
}
|
|
|
|
/// Fetches and fast-forwards, reporting whether anything arrived.
|
|
fn pull(&self, force: bool) -> Result<bool, crate::git::PullError> {
|
|
self.begin_project("fetching");
|
|
crate::git::fetch(&self.project_path, self.git_ipv4)?;
|
|
// Counted after the fetch, from refs now on disk: this is the one
|
|
// place that has the objects to count against.
|
|
let behind = crate::git::behind(&self.project_path);
|
|
if behind == 0 {
|
|
return Ok(false);
|
|
}
|
|
self.begin_project("pulling");
|
|
crate::git::pull(&self.project_path, self.git_ipv4, force)?;
|
|
tracing::info!(
|
|
"pulled {behind} commit(s) into {}",
|
|
self.project_path.display()
|
|
);
|
|
// The card should stop offering what was just taken, rather than
|
|
// waiting out the check interval to notice.
|
|
self.shared.remote_checks.mark_current(&self.project_path);
|
|
Ok(true)
|
|
}
|
|
|
|
/// Builds every component `name` selects, at once.
|
|
///
|
|
/// Every one of them together rather than in turn -- they are
|
|
/// independent, a Rust build and a Gradle build share nothing but the
|
|
/// machine, and measured on this one, running them together takes
|
|
/// about three quarters of the time running them in turn does. The
|
|
/// saving only appears when more than one has work to do, which is
|
|
/// the case a pull produces; `name` is how a caller that wants only
|
|
/// one opts out of paying for the others (see
|
|
/// [`Self::trigger_if_needed`]) -- `None` still means all of them.
|
|
///
|
|
/// What running them together costs is that a failure no longer stops
|
|
/// the others: they are already running by the time it happens, so
|
|
/// stopping them would mean killing work that is probably fine, and
|
|
/// the first failure in declaration order is the one reported, which
|
|
/// is what a walk in that order would have said.
|
|
///
|
|
/// Each component's name is the phase name, so the card says which one
|
|
/// is being worked on without needing anything new to carry it.
|
|
fn run_build(self: &Arc<Self>, name: Option<&str>, record: &RecordBuilt) {
|
|
let mut running = Vec::new();
|
|
for (index, component) in self.components.iter().enumerate() {
|
|
if name.is_some_and(|name| component.name() != name) {
|
|
continue;
|
|
}
|
|
if component.build().is_empty() {
|
|
continue;
|
|
}
|
|
let this = Arc::clone(self);
|
|
let record = Arc::clone(record);
|
|
running.push((
|
|
component.name().to_string(),
|
|
std::thread::spawn(move || this.build_indexed(index, &record)),
|
|
));
|
|
}
|
|
|
|
let mut error = None;
|
|
// Which component's step ended the walk, so the card can open that
|
|
// component's *build* log rather than its runtime one.
|
|
let mut failed = None;
|
|
// Set by the component that is this process; acted on once every
|
|
// other component has finished and the whole run is reported. See
|
|
// `is_self`.
|
|
let mut restart_self = false;
|
|
for (name, handle) in running {
|
|
let outcome = match handle.join() {
|
|
Ok(outcome) => outcome,
|
|
// A panic in a build thread is this server's bug, not the
|
|
// project's, but the card still has to say something --
|
|
// silence would read as a build that simply did nothing.
|
|
Err(_) => Err(format!("building {name} panicked -- see this server's log")),
|
|
};
|
|
match outcome {
|
|
Ok(is_self) => restart_self |= is_self,
|
|
Err(message) if error.is_none() => {
|
|
error = Some(message);
|
|
failed = Some(name);
|
|
}
|
|
Err(_) => {}
|
|
}
|
|
}
|
|
|
|
let mut restart = error.is_none() && restart_self;
|
|
// A build with nothing to do leaves the binary alone, and exec-ing
|
|
// into the same file would drop the phone's connection to deliver
|
|
// the build it already had.
|
|
if restart && crate::restart::binary_unchanged() {
|
|
tracing::info!(
|
|
"the build left this server's binary untouched, so it is not restarting"
|
|
);
|
|
restart = false;
|
|
}
|
|
self.finish(error, failed);
|
|
if restart {
|
|
// Answered and finished first, so whatever happens next cannot
|
|
// take the report away from whoever asked for it.
|
|
crate::restart::deferred(self.handover(), Arc::clone(&self.shared.downloads));
|
|
}
|
|
}
|
|
|
|
/// Builds the component at `index`, and closes its entry whatever
|
|
/// happens.
|
|
///
|
|
/// By index because the components belong to this state, which the
|
|
/// thread holds an `Arc` to -- borrowing one across a thread boundary
|
|
/// would tie the thread's lifetime to this call's.
|
|
///
|
|
/// Answers whether that component is this server itself, which is the
|
|
/// one thing the caller has to act on after every thread is done.
|
|
fn build_indexed(self: &Arc<Self>, index: usize, record: &RecordBuilt) -> Result<bool, String> {
|
|
let component = &self.components[index];
|
|
let outcome = self.build_component(component, record);
|
|
// One place, so no exit from `build_component` can leave a
|
|
// component looking like it is still working.
|
|
self.finish_component(component.name(), outcome.as_ref().err().cloned());
|
|
outcome
|
|
}
|
|
|
|
/// Builds one component and delivers it, or says why it could not.
|
|
///
|
|
/// One function rather than the walk's body inline, so that "the build
|
|
/// stopped here" is recorded in exactly one place -- every failure
|
|
/// leaves by the same return, and the caller names the component
|
|
/// without each site having to remember to.
|
|
fn build_component(
|
|
self: &Arc<Self>,
|
|
component: &Component,
|
|
record: &RecordBuilt,
|
|
) -> Result<bool, String> {
|
|
self.begin_component(component.name(), "building");
|
|
if let Err(message) = self.run_streaming(component) {
|
|
tracing::error!("{} failed: {message}", component.name());
|
|
return Err(message);
|
|
}
|
|
// Recorded per component and only on success, so a walk that
|
|
// fails partway credits what actually built and nothing else:
|
|
// if the APK fails after the server succeeded, the server is
|
|
// current and the APK is not. Scoped to the component's own
|
|
// directory, so a commit touching only the app does not make
|
|
// the server look stale.
|
|
//
|
|
// A checkout that cannot be read records nothing, which reads
|
|
// as unknown rather than as current -- see `component_is_stale`.
|
|
if let Some(sha) = crate::git::subtree_head(&self.project_path, component.cwd()) {
|
|
self.inner
|
|
.lock()
|
|
.unwrap()
|
|
.built_from
|
|
.insert(component.name().to_string(), sha.clone());
|
|
record(component.name(), sha);
|
|
}
|
|
|
|
// Delivering a server is restarting it, so it happens here rather
|
|
// than at the end: a later component's failure should not leave a
|
|
// built-but-not-restarted server behind.
|
|
let Component::Server { name, .. } = component else {
|
|
return Ok(false);
|
|
};
|
|
let Some(service) = crate::service::driver(&self.key, component) else {
|
|
return Ok(false);
|
|
};
|
|
let service = &service;
|
|
|
|
// What the manager says decides both halves: whether the unit is
|
|
// worth refreshing, and whether anything should be restarted
|
|
// afterwards.
|
|
let installed = match crate::service::status(service, &self.project_path, component.cwd()) {
|
|
Ok(state) => Some(state),
|
|
// Not a build failure: the build worked, and whether the
|
|
// service came back is a separate thing the card reports on
|
|
// its own.
|
|
Err(err) => {
|
|
tracing::warn!("could not ask {name} how it is running: {err}");
|
|
None
|
|
}
|
|
};
|
|
|
|
// Refresh the unit whenever there is one, for either kind of
|
|
// server. It is generated from a script in the repository, so a
|
|
// pull can change how the service is *defined* and not only what
|
|
// it runs -- and that change has to land even when the build
|
|
// produced an identical binary, which is the ordinary case for a
|
|
// pull that only touched the service script. `start.sh` has
|
|
// always reinstalled every run for exactly this reason.
|
|
//
|
|
// Safe on a running service: writing the unit and reloading stops
|
|
// nothing.
|
|
if installed.is_some_and(|state| state != crate::service::ServiceState::NotInstalled) {
|
|
self.begin_component(name, "installing");
|
|
if let Err(message) =
|
|
self.run_streaming_command(name, service, component.cwd(), &["install"], None)
|
|
{
|
|
tracing::error!("installing {name} failed: {message}");
|
|
return Err(message);
|
|
}
|
|
}
|
|
|
|
if self.is_self {
|
|
// This one is the process doing the walking, so its restart
|
|
// waits until every other component is done and the result is
|
|
// reported. See the end of `run_build`.
|
|
return Ok(true);
|
|
}
|
|
if installed == Some(crate::service::ServiceState::Running) {
|
|
self.begin_component(name, "restarting");
|
|
if let Err(message) =
|
|
self.run_streaming_command(name, service, component.cwd(), &["restart"], None)
|
|
{
|
|
tracing::error!("restarting {name} failed: {message}");
|
|
return Err(message);
|
|
}
|
|
} else if let Some(state) = installed {
|
|
tracing::info!("{name} is {state:?}, so it was not started");
|
|
}
|
|
Ok(false)
|
|
}
|
|
|
|
/// How this server should be restarted, if it declares a way.
|
|
///
|
|
/// Only ever asked for this server's own project, where exactly one
|
|
/// `Server` component is this process.
|
|
fn handover(&self) -> Option<crate::restart::Handover> {
|
|
let (component, script) = self.components.iter().find_map(|component| {
|
|
matches!(component, Component::Server { .. })
|
|
.then(|| crate::service::driver(&self.key, component))
|
|
.flatten()
|
|
.map(|script| (component, script))
|
|
})?;
|
|
Some(crate::restart::Handover {
|
|
script,
|
|
project: self.project_path.clone(),
|
|
cwd: component.cwd().map(std::path::Path::to_path_buf),
|
|
})
|
|
}
|
|
|
|
/// Runs the build command, recording its output as it arrives rather
|
|
/// than collecting it at the end.
|
|
///
|
|
/// A build is the long part of a pull, and a phone showing nothing but
|
|
/// a spinner for a minute is indistinguishable from one that is stuck.
|
|
/// Reading the pipes as they fill also means a chatty build can't fill
|
|
/// a pipe buffer and block itself, which collecting output at the end
|
|
/// risks for anything verbose.
|
|
fn run_streaming(self: &Arc<Self>, component: &Component) -> Result<(), String> {
|
|
// Opened per component and rotated here, so a failed build is
|
|
// still readable after this server restarts -- the tail kept in
|
|
// memory for the progress display dies with the process, and a
|
|
// build that brought the server down is exactly the one somebody
|
|
// wants to read afterwards.
|
|
let log = crate::logs::open_build_log(&self.key, component.name())
|
|
.map(|file| Arc::new(Mutex::new(file)));
|
|
self.run_streaming_command(
|
|
component.name(),
|
|
component.build(),
|
|
component.cwd(),
|
|
&[],
|
|
log,
|
|
)
|
|
}
|
|
|
|
fn run_streaming_command(
|
|
self: &Arc<Self>,
|
|
component: &str,
|
|
command: &crate::config::Command,
|
|
cwd: Option<&std::path::Path>,
|
|
extra: &[&str],
|
|
log: Option<Arc<Mutex<std::fs::File>>>,
|
|
) -> Result<(), String> {
|
|
let mut child = self
|
|
.spawn(command, cwd, extra)?
|
|
.stdout(std::process::Stdio::piped())
|
|
.stderr(std::process::Stdio::piped())
|
|
.spawn()
|
|
.map_err(|err| format!("failed to start the build: {err}"))?;
|
|
|
|
// Both pipes, on separate threads: whichever the build writes to,
|
|
// waiting on only one of them would deadlock when the other fills.
|
|
let mut readers = Vec::new();
|
|
for pipe in [
|
|
child
|
|
.stdout
|
|
.take()
|
|
.map(|out| Box::new(out) as Box<dyn std::io::Read + Send>),
|
|
child
|
|
.stderr
|
|
.take()
|
|
.map(|err| Box::new(err) as Box<dyn std::io::Read + Send>),
|
|
]
|
|
.into_iter()
|
|
.flatten()
|
|
{
|
|
let state = Arc::clone(self);
|
|
let log = log.clone();
|
|
// Owned per reader thread, because the line it routes has to
|
|
// reach the component that printed it and the thread outlives
|
|
// this borrow.
|
|
let owned = component.to_string();
|
|
readers.push(std::thread::spawn(move || {
|
|
read_segments(pipe, |segment| {
|
|
let segment = segment.trim_end();
|
|
if segment.is_empty() {
|
|
return;
|
|
}
|
|
// The file gets everything, including the progress
|
|
// markers: it is the record of what the command
|
|
// actually printed, not the display. One segment per
|
|
// line, which is already what a wrapper emitting
|
|
// `@@progress` produces, and what keeps a line from
|
|
// one pipe out of the middle of a line from the other.
|
|
if let Some(log) = &log {
|
|
use std::io::Write;
|
|
let _ = writeln!(log.lock().unwrap(), "{segment}");
|
|
}
|
|
match parse_progress(segment) {
|
|
// Machinery, not output: it would only push the
|
|
// build's own last line out of the display.
|
|
Some(progress) => state.set_progress(&owned, progress),
|
|
None => state.log_line(&owned, segment.to_string()),
|
|
}
|
|
});
|
|
}));
|
|
}
|
|
|
|
let status = child
|
|
.wait()
|
|
.map_err(|err| format!("waiting on the build: {err}"))?;
|
|
for reader in readers {
|
|
let _ = reader.join();
|
|
}
|
|
|
|
if status.success() {
|
|
return Ok(());
|
|
}
|
|
// The tail is what a failing build explains itself in, and it is
|
|
// already here rather than needing to be collected separately.
|
|
let tail: Vec<String> = self
|
|
.inner
|
|
.lock()
|
|
.unwrap()
|
|
.runs
|
|
.iter()
|
|
.find(|run| run.name == component)
|
|
.map(|run| run.log.iter().cloned().collect())
|
|
.unwrap_or_default();
|
|
Err(format!(
|
|
"the build step failed ({status}):\n{}",
|
|
truncate_tail(&tail.join("\n"), 2000)
|
|
))
|
|
}
|
|
|
|
/// The command to run, configured but not started.
|
|
fn spawn(
|
|
&self,
|
|
command: &crate::config::Command,
|
|
cwd: Option<&std::path::Path>,
|
|
extra: &[&str],
|
|
) -> Result<Command, String> {
|
|
let mut process = command.to_process(&self.project_path, cwd, extra)?;
|
|
// Cargo hides its progress counter unless it is talking to a
|
|
// terminal, and it is talking to a pipe. Asked for it explicitly,
|
|
// it prints `Building [===> ] 236/237: crate` -- counts it worked
|
|
// out itself, which is the only kind this shows. Set here rather
|
|
// than asked of each project, because a build that happens to be
|
|
// cargo should not have to know that it is being watched, and
|
|
// because a project could otherwise only offer this by
|
|
// reimplementing it.
|
|
//
|
|
// Harmless to anything that is not cargo: an unread variable.
|
|
// Width fixed so the bar is not sized against a terminal there
|
|
// isn't; nothing reads the bar itself, only the numbers after it.
|
|
process.env("CARGO_TERM_PROGRESS_WHEN", "always");
|
|
process.env("CARGO_TERM_PROGRESS_WIDTH", "100");
|
|
// For the build systems that cannot be asked, a wrapper this
|
|
// server ships. Handed over as a path rather than expected on
|
|
// PATH, and absent when a person runs the same script by hand --
|
|
// which is why a script tests for it rather than depending on it.
|
|
// See `crate::shipped`.
|
|
process.env("DEV_UPDATER_PROGRESS", crate::shipped::build_progress());
|
|
Ok(process)
|
|
}
|
|
|
|
/// Marks the start of a step, closing the previous one with its
|
|
/// duration so the phone can show where the time went.
|
|
/// Marks what the *project* is doing. Only fetching and pulling: the
|
|
/// rest belongs to a component.
|
|
fn begin_project(&self, name: &str) {
|
|
self.inner.lock().unwrap().phase = Some(name.to_string());
|
|
}
|
|
|
|
/// Marks one component as doing `step`, starting its entry if this is
|
|
/// the first thing it has done this run.
|
|
fn begin_component(&self, component: &str, step: &str) {
|
|
let mut inner = self.inner.lock().unwrap();
|
|
// The project-wide phase is over once a component is working;
|
|
// otherwise the card shows "pulling" under a build.
|
|
inner.phase = None;
|
|
match inner.runs.iter_mut().find(|run| run.name == component) {
|
|
Some(run) => {
|
|
run.step = Some(step.to_string());
|
|
// A count belongs to the step that printed it.
|
|
run.progress = None;
|
|
}
|
|
None => inner.runs.push(ComponentRun {
|
|
name: component.to_string(),
|
|
step: Some(step.to_string()),
|
|
started: Instant::now(),
|
|
took_ms: None,
|
|
progress: None,
|
|
log: VecDeque::new(),
|
|
error: None,
|
|
}),
|
|
}
|
|
}
|
|
|
|
/// Marks one component as done, with why it stopped if it failed.
|
|
fn finish_component(&self, component: &str, error: Option<String>) {
|
|
let mut inner = self.inner.lock().unwrap();
|
|
if let Some(run) = inner.runs.iter_mut().find(|run| run.name == component) {
|
|
run.took_ms = Some(run.started.elapsed().as_millis() as u64);
|
|
run.step = None;
|
|
run.error = error;
|
|
}
|
|
}
|
|
|
|
/// Closes the current step and ends the run.
|
|
fn finish(&self, error: Option<String>, failed: Option<String>) {
|
|
let mut inner = self.inner.lock().unwrap();
|
|
inner.phase = None;
|
|
inner.building = false;
|
|
// Cleared so the elapsed time belongs to a run in progress rather
|
|
// than counting up forever on an idle server.
|
|
inner.started = None;
|
|
inner.error = error;
|
|
inner.failed = failed;
|
|
}
|
|
|
|
/// Finishes a run that never got past its pull.
|
|
///
|
|
/// Separate from [`Self::finish`] only because a pull failure carries
|
|
/// the one thing a build failure cannot: whether abandoning this
|
|
/// checkout's own history would clear it. Cleared where its siblings
|
|
/// are, at the start of every run, so this is the only thing that can
|
|
/// ever make it true.
|
|
fn fail_pull(&self, error: crate::git::PullError) {
|
|
self.inner.lock().unwrap().unrelated_histories = error.unrelated_histories;
|
|
// No component: a pull fails before the walk starts.
|
|
self.finish(Some(error.message), None);
|
|
}
|
|
|
|
/// Records how far the running command says it has got. Replaces the
|
|
/// previous count rather than accumulating: the command reports its
|
|
/// own total, and `begin` clears this so one component's count is
|
|
/// never shown against the next one's work.
|
|
fn set_progress(&self, component: &str, progress: (u64, u64)) {
|
|
let mut inner = self.inner.lock().unwrap();
|
|
if let Some(run) = inner.runs.iter_mut().find(|run| run.name == component) {
|
|
run.progress = Some(progress);
|
|
}
|
|
}
|
|
|
|
/// Records one line of a component's output for its progress display,
|
|
/// dropping the oldest once the window is full.
|
|
///
|
|
/// Kept against the component that printed it rather than in one
|
|
/// stream, so two building at once do not interleave into something
|
|
/// neither of them said.
|
|
fn log_line(&self, component: &str, line: String) {
|
|
let mut inner = self.inner.lock().unwrap();
|
|
if let Some(run) = inner.runs.iter_mut().find(|run| run.name == component) {
|
|
if run.log.len() == LOG_LINES {
|
|
run.log.pop_front();
|
|
}
|
|
run.log.push_back(line);
|
|
}
|
|
}
|
|
|
|
/// Whether there is anything to run, as opposed to a checkout that
|
|
/// can only be pulled.
|
|
pub fn has_command(&self) -> bool {
|
|
crate::config::has_command(&self.components)
|
|
}
|
|
|
|
/// Whether the last completed build stopped at this component.
|
|
///
|
|
/// The card asks so it can open that component's build log first. A
|
|
/// build that has not failed leaves every component answering false,
|
|
/// which is what makes the runtime log the ordinary default.
|
|
pub fn build_failed(&self, component: &str) -> bool {
|
|
let inner = self.inner.lock().unwrap();
|
|
inner.error.is_some() && inner.failed.as_deref() == Some(component)
|
|
}
|
|
|
|
pub fn status(&self) -> BuildStatus {
|
|
// Before the lock, not inside it -- see `is_stale`.
|
|
let stale = self.is_stale(None);
|
|
let inner = self.inner.lock().unwrap();
|
|
BuildStatus {
|
|
stale,
|
|
building: inner.building,
|
|
error: inner.error.clone(),
|
|
unrelated_histories: inner.unrelated_histories,
|
|
phase: inner.phase.clone(),
|
|
elapsed_ms: inner
|
|
.started
|
|
.map(|started| started.elapsed().as_millis() as u64)
|
|
.unwrap_or(0),
|
|
components: inner
|
|
.runs
|
|
.iter()
|
|
.map(|run| ComponentStatus {
|
|
name: run.name.clone(),
|
|
step: run.step.clone(),
|
|
// Still going, or however long it took.
|
|
elapsed_ms: run
|
|
.took_ms
|
|
.unwrap_or_else(|| run.started.elapsed().as_millis() as u64),
|
|
progress: run
|
|
.progress
|
|
.filter(|(_, total)| *total > 0)
|
|
.map(|(done, total)| Progress {
|
|
done: done.min(total),
|
|
total,
|
|
}),
|
|
log: run.log.iter().cloned().collect(),
|
|
error: run.error.clone(),
|
|
})
|
|
.collect(),
|
|
}
|
|
}
|
|
}
|
|
|
|
fn truncate_tail(s: &str, max_chars: usize) -> String {
|
|
let char_count = s.chars().count();
|
|
if char_count <= max_chars {
|
|
s.to_string()
|
|
} else {
|
|
s.chars().skip(char_count - max_chars).collect()
|
|
}
|
|
}
|
|
|
|
#[cfg(test)]
|
|
mod tests {
|
|
/// Cargo's own counter, which `spawn` asks it to print into a pipe.
|
|
/// The numbers are cargo's; nothing here works them out.
|
|
#[test]
|
|
fn cargos_progress_counter_is_read_and_other_numbers_are_not() {
|
|
assert_eq!(
|
|
parse_progress(" Building [=====> ] 236/237: dev-updater(bin)"),
|
|
Some((236, 237))
|
|
);
|
|
assert_eq!(parse_progress(" Building [] 0/12"), Some((0, 12)));
|
|
|
|
// Build output is full of numbers; only the bracketed bar means a
|
|
// count. A crate that happens to be called this must not move it.
|
|
assert_eq!(parse_progress(" Compiling foo v1.2.3 (12/34)"), None);
|
|
assert_eq!(parse_progress("warning: 3/4 of these are unused"), None);
|
|
assert_eq!(parse_progress("Building the thing 1/2"), None);
|
|
}
|
|
|
|
/// A project can still report its own, which is the only way a tool
|
|
/// with no counter of its own gets a bar.
|
|
#[test]
|
|
fn the_explicit_marker_still_wins() {
|
|
assert_eq!(parse_progress("@@progress 3/7"), Some((3, 7)));
|
|
assert_eq!(parse_progress("@@progress nonsense"), None);
|
|
}
|
|
|
|
use std::path::Path;
|
|
|
|
use super::*;
|
|
|
|
fn run(dir: &Path, args: &[&str]) {
|
|
let out = std::process::Command::new(args[0])
|
|
.args(&args[1..])
|
|
.current_dir(dir)
|
|
.output()
|
|
.expect("run");
|
|
assert!(
|
|
out.status.success(),
|
|
"{args:?}: {}",
|
|
String::from_utf8_lossy(&out.stderr)
|
|
);
|
|
}
|
|
|
|
/// One checkout producing two things, which is the shape both real
|
|
/// projects have: a server under `server/` and an app under `app/`.
|
|
fn two_component_checkout(root: &Path) -> Vec<Component> {
|
|
std::fs::create_dir_all(root.join("server")).expect("mkdir");
|
|
std::fs::create_dir_all(root.join("app/build/outputs/apk/debug")).expect("mkdir");
|
|
// Something for `find_apks` to see, so staleness is decided by the
|
|
// rules under test rather than by nothing being built.
|
|
std::fs::write(root.join("app/build/outputs/apk/debug/a.apk"), b"").expect("write");
|
|
std::fs::write(root.join("server/main.rs"), "one").expect("write");
|
|
std::fs::write(root.join("app/main.kt"), "one").expect("write");
|
|
for args in [
|
|
&["git", "init", "-q", "-b", "main"][..],
|
|
&["git", "config", "user.email", "t@example.com"],
|
|
&["git", "config", "user.name", "Test"],
|
|
&["git", "add", "-A"],
|
|
&["git", "commit", "-qm", "one"],
|
|
] {
|
|
run(root, args);
|
|
}
|
|
vec![
|
|
Component::Server {
|
|
name: "backend".to_string(),
|
|
build: crate::config::Command::from_line("true"),
|
|
cwd: Some(PathBuf::from("server")),
|
|
stale_when: None,
|
|
service: None,
|
|
built_from: None,
|
|
},
|
|
Component::Apk {
|
|
name: "app".to_string(),
|
|
build: crate::config::Command::from_line("true"),
|
|
cwd: Some(PathBuf::from("app")),
|
|
stale_when: None,
|
|
strip: false,
|
|
package: None,
|
|
built_from: None,
|
|
},
|
|
]
|
|
}
|
|
|
|
/// A pull that brings a changed declaration must stop, not build.
|
|
///
|
|
/// The gate answers "may this project's command run?", and the
|
|
/// declaration it reads lives in the checkout -- so the pull is the
|
|
/// one thing that changes the answer. Asked before the fast-forward
|
|
/// it necessarily reports on the commit being replaced, which is how
|
|
/// a card went off and updated the app instead of stopping to ask.
|
|
/// Hence a closure: this holds the *moment* it is called.
|
|
#[tokio::test]
|
|
async fn a_pull_that_changes_the_declaration_does_not_build() {
|
|
let dir = tempfile::tempdir().expect("tempdir");
|
|
let origin = dir.path().join("origin");
|
|
std::fs::create_dir_all(&origin).expect("mkdir");
|
|
let declare =
|
|
|command: &str| format!("components: [Apk(name: \"app\", build: \"{command}\")],\n");
|
|
std::fs::write(
|
|
origin.join(crate::config::PROJECT_CONFIG_FILE),
|
|
declare("touch built-marker"),
|
|
)
|
|
.expect("write");
|
|
for args in [
|
|
&["git", "init", "-q", "-b", "main"][..],
|
|
&["git", "config", "user.email", "t@example.com"],
|
|
&["git", "config", "user.name", "Test"],
|
|
&["git", "add", "-A"],
|
|
&["git", "commit", "-qm", "one"],
|
|
] {
|
|
run(&origin, args);
|
|
}
|
|
let clone = dir.path().join("clone");
|
|
run(
|
|
dir.path(),
|
|
&[
|
|
"git",
|
|
"clone",
|
|
"-q",
|
|
origin.to_str().unwrap(),
|
|
clone.to_str().unwrap(),
|
|
],
|
|
);
|
|
|
|
// What this machine accepted, which is what the checkout asks for
|
|
// right now -- so the gate is open at the moment the button is
|
|
// pressed, and only the pull closes it.
|
|
let accepted = vec![Component::Apk {
|
|
name: "app".to_string(),
|
|
build: crate::config::Command::from_line("touch built-marker"),
|
|
cwd: None,
|
|
stale_when: None,
|
|
strip: false,
|
|
package: None,
|
|
built_from: None,
|
|
}];
|
|
let state = state_for(&clone, accepted.clone());
|
|
|
|
// The commit the pull is about to take, asking for a different
|
|
// command from the one that was accepted.
|
|
std::fs::write(
|
|
origin.join(crate::config::PROJECT_CONFIG_FILE),
|
|
declare("touch never-run"),
|
|
)
|
|
.expect("write");
|
|
run(&origin, &["git", "commit", "-qam", "two"]);
|
|
|
|
let project = clone.clone();
|
|
state.pull_and_build(
|
|
false,
|
|
move || crate::config::project_config(&project).matches_accepted(&accepted, None),
|
|
Arc::new(|_: &str, _: String| {}),
|
|
);
|
|
while state.status().building {
|
|
tokio::time::sleep(std::time::Duration::from_millis(10)).await;
|
|
}
|
|
|
|
assert!(
|
|
clone.join(".dev-updater.ron").exists(),
|
|
"the pull itself still happens -- it is how the new request arrives to be read",
|
|
);
|
|
assert!(
|
|
!clone.join("built-marker").exists(),
|
|
"the declaration changed under the acceptance, so nothing may run",
|
|
);
|
|
}
|
|
|
|
/// Naming one component builds only that one.
|
|
///
|
|
/// The case a project with more than one component exists to avoid
|
|
/// paying for: pressing Update on one client of a two-client project
|
|
/// must not also run the other's build, which for a slow one (an ARM
|
|
/// cross-compile, say) is the entire complaint a phone would have.
|
|
#[tokio::test]
|
|
async fn naming_a_component_builds_only_that_one() {
|
|
let dir = tempfile::tempdir().expect("tempdir");
|
|
let root = dir.path();
|
|
std::fs::create_dir_all(root.join("server")).expect("mkdir");
|
|
std::fs::create_dir_all(root.join("app")).expect("mkdir");
|
|
let components = vec![
|
|
Component::Server {
|
|
name: "backend".to_string(),
|
|
build: crate::config::Command::from_line("touch backend-built"),
|
|
cwd: Some(PathBuf::from("server")),
|
|
stale_when: None,
|
|
service: None,
|
|
built_from: None,
|
|
},
|
|
Component::Apk {
|
|
name: "app".to_string(),
|
|
build: crate::config::Command::from_line("touch app-built"),
|
|
cwd: Some(PathBuf::from("app")),
|
|
stale_when: None,
|
|
strip: false,
|
|
package: None,
|
|
built_from: None,
|
|
},
|
|
];
|
|
let state = state_for(root, components);
|
|
|
|
state.build_now(Some("app"), Arc::new(|_: &str, _: String| {}));
|
|
let deadline = std::time::Instant::now() + std::time::Duration::from_secs(10);
|
|
while state.status().building {
|
|
assert!(
|
|
std::time::Instant::now() < deadline,
|
|
"build did not finish in time"
|
|
);
|
|
tokio::time::sleep(std::time::Duration::from_millis(10)).await;
|
|
}
|
|
|
|
assert!(
|
|
root.join("app/app-built").exists(),
|
|
"the named component ran"
|
|
);
|
|
assert!(
|
|
!root.join("server/backend-built").exists(),
|
|
"naming one component must not build the other",
|
|
);
|
|
|
|
let status = state.status();
|
|
assert_eq!(
|
|
status
|
|
.components
|
|
.iter()
|
|
.map(|c| c.name.as_str())
|
|
.collect::<Vec<_>>(),
|
|
vec!["app"],
|
|
"the status only reports the component that actually ran",
|
|
);
|
|
}
|
|
|
|
/// A sibling that has already been built must not mask that this one
|
|
/// never has.
|
|
///
|
|
/// The bug the previous test's fix uncovered: `APK_PATTERNS` includes
|
|
/// a one-level pattern (`*/build/outputs/apk/*/*.apk`), so a second
|
|
/// component's output sitting one directory below the project root
|
|
/// satisfies a scan of the *root* even when it belongs to a component
|
|
/// nobody has ever built here. `component_is_stale`'s "never built"
|
|
/// check used to run against the project root rather than the
|
|
/// component's own directory, so the moment either of two components
|
|
/// had been built once, the other quietly stopped being offered its
|
|
/// own first build -- `prepare` (what Update runs before a download)
|
|
/// saw a component with a command and no output and declared it
|
|
/// current anyway.
|
|
#[tokio::test]
|
|
async fn a_sibling_already_built_does_not_hide_that_this_one_never_has() {
|
|
let dir = tempfile::tempdir().expect("tempdir");
|
|
let root = dir.path();
|
|
std::fs::create_dir_all(root.join("a")).expect("mkdir");
|
|
std::fs::create_dir_all(root.join("b")).expect("mkdir");
|
|
let components = vec![
|
|
Component::Apk {
|
|
name: "a".to_string(),
|
|
build: crate::config::Command::from_line("touch a-built"),
|
|
cwd: Some(PathBuf::from("a")),
|
|
stale_when: None,
|
|
strip: false,
|
|
package: None,
|
|
built_from: None,
|
|
},
|
|
Component::Apk {
|
|
name: "b".to_string(),
|
|
build: crate::config::Command::from_line("touch b-built"),
|
|
cwd: Some(PathBuf::from("b")),
|
|
stale_when: None,
|
|
strip: false,
|
|
package: None,
|
|
built_from: None,
|
|
},
|
|
];
|
|
let state = state_for(root, components);
|
|
|
|
// `a` has an APK on disk already -- built by something other than
|
|
// this state, which is the ordinary case for "added after the
|
|
// fact" -- while `b` has never been built at all.
|
|
std::fs::create_dir_all(root.join("a/build/outputs/apk/debug")).expect("mkdir");
|
|
std::fs::write(root.join("a/build/outputs/apk/debug/a.apk"), b"").expect("write");
|
|
|
|
assert!(
|
|
stale(&state, "b"),
|
|
"b has never been built, regardless of what a has on disk",
|
|
);
|
|
|
|
state.trigger_if_needed(Some("b"), Arc::new(|_: &str, _: String| {}));
|
|
let deadline = std::time::Instant::now() + std::time::Duration::from_secs(10);
|
|
while state.status().building {
|
|
assert!(
|
|
std::time::Instant::now() < deadline,
|
|
"build did not finish in time"
|
|
);
|
|
tokio::time::sleep(std::time::Duration::from_millis(10)).await;
|
|
}
|
|
assert!(
|
|
root.join("b/b-built").exists(),
|
|
"prepare must have run b's build"
|
|
);
|
|
}
|
|
|
|
fn state_for(root: &Path, components: Vec<Component>) -> Arc<BuildState> {
|
|
BuildState::new(
|
|
"test".to_string(),
|
|
components,
|
|
true,
|
|
false,
|
|
root.to_path_buf(),
|
|
Shared {
|
|
remote_checks: crate::git::RemoteChecks::default(),
|
|
downloads: Arc::new(crate::restart::Downloads::default()),
|
|
},
|
|
false,
|
|
)
|
|
}
|
|
|
|
fn stale(state: &BuildState, name: &str) -> bool {
|
|
let component = state
|
|
.components
|
|
.iter()
|
|
.find(|component| component.name() == name)
|
|
.expect("component");
|
|
state.component_is_stale(component, &state.inner.lock().unwrap().built_from.clone())
|
|
}
|
|
|
|
/// The shape of bug a unit test that only calls `component_is_stale`
|
|
/// cannot see: `status()` and `trigger_if_needed` hold the lock that
|
|
/// the staleness check also wants. Taking it twice on one thread is a
|
|
/// deadlock, and it hung every `/status` and `/prepare` for any
|
|
/// project that had a build on disk -- which is every real one.
|
|
///
|
|
/// Run on a thread with a deadline so a regression fails in
|
|
/// milliseconds instead of hanging the suite.
|
|
#[test]
|
|
fn asking_for_status_does_not_deadlock() {
|
|
let dir = tempfile::tempdir().expect("tempdir");
|
|
let root = dir.path();
|
|
let components = two_component_checkout(root);
|
|
let state = state_for(root, components);
|
|
|
|
let (tx, rx) = std::sync::mpsc::channel();
|
|
std::thread::spawn(move || {
|
|
let _ = state.status();
|
|
let _ = tx.send(());
|
|
});
|
|
assert!(
|
|
rx.recv_timeout(std::time::Duration::from_secs(5)).is_ok(),
|
|
"status() deadlocked on its own lock",
|
|
);
|
|
}
|
|
|
|
/// A commit that touches one component's directory must not make the
|
|
/// other look behind. Without the scoping, every app-only change would
|
|
/// offer a pointless rebuild of the server -- in the one repository
|
|
/// shape this feature was asked for.
|
|
#[test]
|
|
fn a_commit_in_one_subtree_leaves_the_other_current() {
|
|
let dir = tempfile::tempdir().expect("tempdir");
|
|
let root = dir.path();
|
|
let components = two_component_checkout(root);
|
|
let state = state_for(root, components);
|
|
|
|
// Built now, from what is checked out now.
|
|
for name in ["backend", "app"] {
|
|
let cwd = PathBuf::from(if name == "backend" { "server" } else { "app" });
|
|
let sha = crate::git::subtree_head(root, Some(&cwd)).expect("a commit");
|
|
state
|
|
.inner
|
|
.lock()
|
|
.unwrap()
|
|
.built_from
|
|
.insert(name.to_string(), sha);
|
|
}
|
|
assert!(!stale(&state, "backend"), "just built");
|
|
assert!(!stale(&state, "app"), "just built");
|
|
|
|
// A change to the app only.
|
|
std::fs::write(root.join("app/main.kt"), "two").expect("write");
|
|
run(root, &["git", "commit", "-qam", "app only"]);
|
|
|
|
assert!(stale(&state, "app"), "the app moved and was not rebuilt");
|
|
assert!(
|
|
!stale(&state, "backend"),
|
|
"the server's directory did not change"
|
|
);
|
|
}
|
|
|
|
/// Nothing recorded is "we have never built this", not "this is out of
|
|
/// date" -- otherwise every project that existed before this feature
|
|
/// would demand a rebuild the moment it arrived.
|
|
#[test]
|
|
fn a_component_this_server_has_never_built_is_not_called_stale() {
|
|
let dir = tempfile::tempdir().expect("tempdir");
|
|
let root = dir.path();
|
|
let components = two_component_checkout(root);
|
|
let state = state_for(root, components);
|
|
|
|
assert!(
|
|
state.inner.lock().unwrap().built_from.is_empty(),
|
|
"nothing recorded"
|
|
);
|
|
assert!(!stale(&state, "backend"));
|
|
assert!(!stale(&state, "app"));
|
|
}
|
|
|
|
/// The same must hold where there is no checkout to compare against:
|
|
/// unreadable is not evidence of anything.
|
|
#[test]
|
|
fn a_project_outside_git_is_not_called_stale() {
|
|
let dir = tempfile::tempdir().expect("tempdir");
|
|
let root = dir.path();
|
|
std::fs::create_dir_all(root.join("app/build/outputs/apk/debug")).expect("mkdir");
|
|
std::fs::write(root.join("app/build/outputs/apk/debug/a.apk"), b"").expect("write");
|
|
let state = state_for(
|
|
root,
|
|
vec![Component::Apk {
|
|
name: "app".to_string(),
|
|
build: crate::config::Command::from_line("true"),
|
|
cwd: None,
|
|
stale_when: None,
|
|
strip: false,
|
|
package: None,
|
|
built_from: Some("0000000000000000000000000000000000000000".to_string()),
|
|
}],
|
|
);
|
|
assert!(!stale(&state, "app"), "no checkout means no opinion");
|
|
}
|
|
|
|
/// The marker is recognised only in exactly its shape; anything else on
|
|
/// a line is the build talking and belongs in the log, where a build
|
|
/// that prints a malformed report can be seen to have done so.
|
|
#[test]
|
|
fn progress_is_read_only_from_the_marker() {
|
|
assert_eq!(parse_progress("@@progress 12/43"), Some((12, 43)));
|
|
assert_eq!(parse_progress(" @@progress 0/1 "), Some((0, 1)));
|
|
assert_eq!(parse_progress("@@progress 7 / 9 "), Some((7, 9)));
|
|
|
|
assert_eq!(parse_progress("Building @@progress 1/2"), None);
|
|
assert_eq!(parse_progress("@@progress soon"), None);
|
|
assert_eq!(parse_progress("@@progress 1/"), None);
|
|
assert_eq!(parse_progress("> Task :app:compileDebugKotlin"), None);
|
|
}
|
|
|
|
/// Feeds `read_segments` one chunk per `read`, recording how many
|
|
/// segments had been emitted before each one was handed over.
|
|
struct Chunks {
|
|
chunks: std::vec::IntoIter<&'static [u8]>,
|
|
emitted: std::rc::Rc<std::cell::RefCell<Vec<String>>>,
|
|
before_each_read: std::rc::Rc<std::cell::RefCell<Vec<usize>>>,
|
|
}
|
|
|
|
impl std::io::Read for Chunks {
|
|
fn read(&mut self, buffer: &mut [u8]) -> std::io::Result<usize> {
|
|
self.before_each_read
|
|
.borrow_mut()
|
|
.push(self.emitted.borrow().len());
|
|
let Some(chunk) = self.chunks.next() else {
|
|
return Ok(0);
|
|
};
|
|
assert!(chunk.len() <= buffer.len(), "the test's chunks are small");
|
|
buffer[..chunk.len()].copy_from_slice(chunk);
|
|
Ok(chunk.len())
|
|
}
|
|
}
|
|
|
|
/// The defect this splitting exists for, and the half of it a batch
|
|
/// test cannot see: it is not whether the counter is *parsed* but
|
|
/// **when** it is delivered.
|
|
///
|
|
/// Cargo ends its counter with a carriage return and no newline, so a
|
|
/// reader waiting for `\n` holds it until whatever prints the next
|
|
/// one. Here that is the `Finished` line, which on a real incremental
|
|
/// build arrives only when the build is over -- so the card had
|
|
/// nothing to measure for the whole compile, which is exactly what a
|
|
/// pull produces. The assertion is therefore about the count of
|
|
/// segments already emitted when the *second* chunk is asked for.
|
|
#[test]
|
|
fn a_carriage_return_delivers_its_segment_without_waiting_for_a_newline() {
|
|
// Byte for byte what `cargo build --release` printed with one
|
|
// dirty crate, shortened only in the bar and the path.
|
|
let chunks: Vec<&'static [u8]> = vec![
|
|
b" Compiling dev-updater v0.1.0 (/p)\n Building [==> ] 236/237: dev-updater(bin) \r",
|
|
b" Finished `release` profile [optimized] target(s) in 7.60s\n",
|
|
];
|
|
let emitted = std::rc::Rc::new(std::cell::RefCell::new(Vec::new()));
|
|
let before_each_read = std::rc::Rc::new(std::cell::RefCell::new(Vec::new()));
|
|
|
|
let collected = std::rc::Rc::clone(&emitted);
|
|
read_segments(
|
|
Chunks {
|
|
chunks: chunks.into_iter(),
|
|
emitted: std::rc::Rc::clone(&emitted),
|
|
before_each_read: std::rc::Rc::clone(&before_each_read),
|
|
},
|
|
|segment| collected.borrow_mut().push(segment.to_string()),
|
|
);
|
|
|
|
assert_eq!(
|
|
before_each_read.borrow().as_slice(),
|
|
[0, 2, 3],
|
|
"the counter must be delivered with its own chunk, not held \
|
|
until the newline in the next one"
|
|
);
|
|
let emitted = emitted.borrow();
|
|
assert_eq!(emitted.len(), 3);
|
|
assert_eq!(
|
|
parse_progress(&emitted[1]),
|
|
Some((236, 237)),
|
|
"and it is still the cargo counter once split"
|
|
);
|
|
}
|
|
|
|
/// The case the fix was not written for: output that was already
|
|
/// arriving line by line must not change shape. A cold build is
|
|
/// mostly this, and it is where a wrapper's `@@progress` lines live
|
|
/// too.
|
|
#[test]
|
|
fn newline_terminated_output_is_unchanged_and_blank_segments_are_dropped() {
|
|
let mut segments = Vec::new();
|
|
read_segments(
|
|
&b"> Task :app:compileDebugKotlin\n@@progress 7/9\r\n\nlast, unterminated"[..],
|
|
|segment| segments.push(segment.to_string()),
|
|
);
|
|
assert_eq!(
|
|
segments,
|
|
[
|
|
"> Task :app:compileDebugKotlin",
|
|
"@@progress 7/9",
|
|
"",
|
|
"",
|
|
"last, unterminated"
|
|
],
|
|
"a CRLF ends one segment and opens an empty one; the caller \
|
|
drops the blanks"
|
|
);
|
|
}
|
|
}
|