The per-column check loops (QC-07/08/09/10/11/13/15, SC-01/02, and the
comparison checks CP-04/05/06/07) no longer grow their result list with
c(results, list(...)) inside a for loop, which reallocated the whole list
on every column and made a run O(columns^2). They now build results with
lapply() and compact once, so wide deliveries (100+ columns) build their
results in linear time. Check output is unchanged.
The missing/empty predicate (is.na(x) | x == "") and the four
comparison-check default thresholds are each now defined once and shared by
the QC checks, the comparison checks, the drift report, and the snapshot
writer, instead of being reimplemented at each call site. This removes the
risk of, for example, the QC report and the drift report applying different
default thresholds to the same rule after only one copy was edited.
The shared Quarto render pipeline (render into a temp dir, verify an output
file was produced, abort with dqcheckr_render_error if not, then move the
result into place) is now a single internal helper (.quarto_render_to_file())
called by both the main report writer and the drift report writer, instead of
being duplicated near-verbatim in each. A future change to the render pipeline
now lands in one place rather than needing to be mirrored by hand.
fwf_widths (must abort with dqcheckr_invalid_config), and
render_report() when Quarto returns without writing an output file (must
abort with dqcheckr_render_error rather than name a report that does not
exist).The CP-04 numeric-mean-shift comparison no longer aborts the whole run when a
numeric column contains a literal Inf/-Inf value. as.numeric("Inf") is
Inf (not NA), so such a column still classifies as numeric and reached the
mean comparison; a non-finite mean slipped past the guard and made the shift
NaN, which crashed the pipeline at if (shift_pct > threshold). The check now
filters to finite values first (as check_outliers() and compute_col_stats()
already do), so an Inf-bearing column is handled as a warning rather than a
crash. R's own write.csv() emits the literal Inf, so this is a realistic
corrupted-delivery input.
compare_snapshots() now rejects a NULL or empty dataset_name with a clear
dqcheckr_invalid_argument error instead of aborting with an empty message. A
NULL bound as SQL NULL (matching no rows) and the "need 2 snapshots" abort
then collapsed to a zero-length, blank-text condition.
read_recent_snapshots() now clamps a negative n to 0 rather than returning
the entire history. SQLite reads LIMIT -1 as "no limit", so a negative n
silently returned every row instead of capping the result as documented.
compare_snapshots() now returns the rendered drift report's path as a
report_path element on its result (NULL when no report was written), so a
caller can link to the report directly instead of reconstructing the filename
from a slug pattern -- the reconstruction approach is fragile and broke when
the drift filename gained its snapshot ids.
A snapshot's render_status now carries a "pending" state while its report
is still rendering. The row is written as "pending" and only flipped to
"success" (with report_file) once the report is confirmed on disk, or to
"failed" if the render is skipped or errors. Previously the row was written
as "success" with a NULL report_file before the render finished, which a
concurrent reader could not tell apart from a completed pre-0.2.3 row and would
turn into a broken report link. Consumers linking to a report should treat a
"pending" row as not-yet-available rather than reconstructing a filename.
The drift report no longer silently reports a report that was never written.
When Quarto returned without error but produced no output file,
compare_snapshots() still announced (and, with open_report = TRUE, tried to
open) the intended path. The drift writer now raises the same typed error the
main report already did, and compare_snapshots() downgrades it to a warning
with no report link -- the computed drift is still returned.
Drift report filenames now include both compared snapshot ids
(drift_dataset_20260718_010203_1_2.html), so two comparisons of one dataset
started in the same wall-clock second (a loop over id pairs, two GUI users) no
longer collide on a single filename and silently overwrite each other -- the
same protection report_filename() already gives the main report.
Column statistics can no longer store a non-finite value even when the
aggregate itself overflows. An earlier fix excluded non-finite inputs
(Inf/-Inf from a corrupted delivery), but sd() of finite-but-very-large
values still overflows the double range internally and returned Inf, which
was stored as the literal string "Inf" and poisoned drift arithmetic. Every
numeric aggregate is now serialised through a finiteness guard (non-finite ->
NA), and the drift reader maps any non-finite value parsed from an older
database to NA as well, so a snapshot written before this fix cannot
re-poison a comparison.
Report filenames now include the snapshot id
(dataset_20260718_010203_47.html), so two runs of one dataset that start in
the same wall-clock second no longer collide on a single filename. Previously
the second run silently overwrote the first run's report while the first run's
snapshot kept pointing at it. The filename is now written to the snapshot's
report_file column by an update after the report is confirmed rendered,
rather than optimistically at insert time, so the column can never name a
report that was not written.
Configuration lookups no longer partial-match. R's $ operator falls back to
a prefix match when the exact element is absent, so a config with no
column_rules: key but a parked or mistyped one (column_rules_disabled,
column_rules_old, ...) had that section silently drive per-column
thresholds -- producing wrong PASS/FAIL verdicts from a section the config
did not contain. Every config-key access now uses exact [[ ]] indexing, so
a parked or renamed section is correctly treated as absent.
QC-16 no longer reports a spurious clean pass for a delivery whose encoding it
did not actually verify. A declared multi-byte or unknown encoding
(UTF-16LE, UTF-32, Shift-JIS, GB18030, ...) used to be reported as
"a single-byte encoding; every byte is valid by construction" -- which is
false, and which also silently disabled the crash guard that scanning gives
UTF-8 deliveries. Such an encoding is now read as declared but reported as a
WARN stating it was not validity-checked. Genuine single-byte encodings
(ISO-8859-x, Windows-125x) still PASS.
The UTF-8 validity scan now streams the file in bounded chunks instead of reading it into one in-memory vector, so an arbitrarily large delivery is verified in flat memory rather than exhausting it on the network-share hosts dqcheckr deploys to.
Non-finite values in a numeric column (Inf/-Inf, e.g. from a corrupted
upstream delivery whose CSV contains the literal text "Inf") are no longer
mishandled. compute_col_stats() excludes them from the mean, standard
deviation, minimum, and maximum, so the snapshot database can no longer store
the literal string "NaN"/"Inf" and poison later drift comparisons; the finite
mean shift a contaminated delivery causes is still reported as drift.
check_outliers() (QC-15), which previously aborted with an uninformative
"missing value where TRUE/FALSE needed" on such a column, now runs cleanly.
A zero-column delivery (for example an empty file) is now recorded. Previously
compute_col_stats() returned NULL, and the snapshot write then failed and
lost the entire run; the run is now snapshotted with a column count of zero.
Failures that were previously swallowed into silent, benign-looking results are now surfaced.
read_recent_snapshots() and list_snapshots() no longer report a
corrupt, locked, or unreadable database as an empty history. They emit a
warning naming the cause before returning the empty frame, so "the read
failed" is distinguishable from "no runs yet".run_dq_check() now appends
"[snapshot NOT recorded]" to its result line instead of printing an
unqualified success while the history row was lost.Concurrent runs sharing one snapshot database no longer lose a snapshot or die
during migration. Connections now set PRAGMA busy_timeout, so a run that
meets a database another run is writing waits for the lock instead of failing
instantly (its snapshot was previously swallowed into a warning). Schema
creation and the auto-migration of older databases now run inside a single
BEGIN IMMEDIATE transaction, so two first-runs after an upgrade can no
longer both add the same column (the loser used to die with "duplicate column
name"). WAL journalling is intentionally not used, as it is unsafe on the
network file systems dqcheckr is deployed on.
init_snapshot_db() no longer modifies a snapshots table it did not create.
If the target database already contains an unrelated table of that name, it
now aborts with a typed dqcheckr_schema_error instead of altering the user's
table. Column-existence checks during migration are now case-insensitive,
matching SQLite, so a column stored as e.g. Report_File is recognised rather
than re-added.
The drift report now flags a missing-rate or non-numeric-rate change in the
same direction as the corresponding comparison check. Both .compute_drift()
columns used abs(), so a column that improved between deliveries (fewer
missing values, or less non-numeric junk) was flagged as breaching, while the
CP-03 and CP-07 checks -- reading the same max_missing_rate_change_pp /
max_non_numeric_rate_change_pp thresholds -- passed it. Drift now breaches
only on an increase, matching the checks. (The numeric-mean-shift drift keeps
its two-directional test, matching the two-directional CP-04 check.)
A run whose HTML report is not produced no longer records a successful-looking
snapshot. When the Quarto CLI is absent the report is skipped with a warning,
but the snapshot row previously kept render_status = "success" and a
report_file naming a report that was never written, so history readers (and
the GUI's "Open report" link) pointed at a file that 404s. The snapshot is now
reconciled against what actually happened: if no report file exists the row is
marked render_status = "failed" and its report_file is cleared. Rendering
that returns without producing a file now raises instead of reporting success,
and report_file is guaranteed to name a report that exists.
Column type inference no longer misclassifies a value whose prefix happens
to be a date. as.Date() matches a prefix and silently ignores trailing
characters, so "2024-01-15x" and the 9-digit id "202401159" (whose first
eight digits parse under %Y%m%d) were both classified as "date" — which
made corrupt dates pass QC-06 and stripped the numeric checks (QC-07/08/11)
from id columns of eight or more digits. Each date format is now gated on an
anchored shape before calendar validation. The documented caveat is
unchanged: a genuine eight-digit value that is also a valid %Y%m%d date
still classifies as a date; a nine-digit id now correctly classifies as
numeric.
read_recent_snapshots() now returns the full set of columns even for a
snapshot database created before 0.2.3. Previously it ran SELECT * without
migrating, so an older database returned rows with the newer columns (such as
report_file) absent rather than NA, and callers relying on those
columns errored instead of degrading. The missing columns are now filled in
after the read with the same defaults a migration would apply; the database
file itself is not modified, so reads remain safe on read-only or
network-shared databases.
starwars_csv demonstration data (inst/demonstrations/data/)
is now shipped. An unanchored .gitignore rule had excluded it, so a fresh
clone was missing the file that 27 example blocks and two tests depend on,
and R CMD check failed on a clean checkout.starwars_fwf) now has its data file. A new
inst/demonstrations/makedata.R derives starwars.fwf from the CSV, so the
FWF half of demo.R runs end-to-end instead of aborting on a missing file.encoding of ASCII (or a formal alias such as US-ASCII) is now
read as UTF-8. ASCII is a strict subset of UTF-8, so this is lossless — and
it removes a hard R session crash ("Invalid multibyte sequence" inside
vroom/iconv, not a catchable R error) when a delivery declared ASCII
contains a byte above 127 beyond whatever sample an encoding sniffer
originally looked at.read_dataset() now validity-scans the entire file before parsing. A
delivery that is not valid UTF-8 no longer risks crashing or silently
producing mojibake: it is read with a single-byte fallback encoding, the
run completes, and QC-16 reports a FAIL naming the detector's best guess at
the actual encoding (suppliers can change export encodings between
deliveries, so this is checked per delivery). Valid files and declared
single-byte encodings (which have no invalid byte sequences) report PASS.
New dependency: stringi.run_dq_check() with an untyped "missing value where TRUE/FALSE needed"
error. Missing rates are defined as 0 for empty inputs, and QC-14 gains an
unconditional "Empty file" FAIL sub-check so an empty delivery always fails
the run — with a snapshot and report — instead of crashing it.detect_files() no longer considers subdirectories of folder when
picking the current/previous file by modification time.dq_result(threshold = NULL) now yields an NA threshold instead of
failing with "argument is of length zero"; invalid vector status values
produce a clear typed error.column_rules pattern now produces a QC-13 FAIL
result naming the pattern instead of aborting the run; an invalid
column_order_severity is rejected at load_config() time with a typed
dqcheckr_invalid_config error instead of aborting mid-check.run_custom_checks() now validates each returned element (required fields
and a valid status) and aborts with dqcheckr_invalid_custom_checks naming
the offending element, instead of failing later with misleading errors.snapshots row without its column_snapshots stats;
column-level custom results are batch-inserted.run_timestamp and the report filename are now derived from
one timestamp taken at the start of the run. Previously they came from two
separate clock reads, so links that reconstruct the report filename from
the snapshot timestamp (as the GUI does) could intermittently point at a
file with a name one second off.compare_snapshots() no longer announces (and offers to open) a drift
report path when Quarto is unavailable and no report was written.file.rename() fails across filesystems (e.g. a
network-share report directory); previously the move failed silently."2.10" matches an allowed value of 2.1.read_recent_snapshots()'s empty-database fallback now has the same
columns as the live query (it was missing comparison_mode,
render_status, and type_changed_cols_vs_previous).report_file column in the snapshots table stores the rendered
report's filename outright (auto-migrated into existing databases), so
consumers no longer need to reconstruct it from the run timestamp.
read_recent_snapshots() returns it; NA for pre-0.2.3 rows.run_dq_check() and compare_snapshots() now document that relative
snapshot_db / report_output_dir paths resolve against the working
directory, not config_dir.infer_col_type() documents its date-format precedence, the
all-must-parse rule, and the all-8-digit-identifier caveat, with a pointer
to column_types overrides.run_qc_checks() and the individual check functions
gain an optional types argument; previously each of QC-06/07/08/11/15,
the column statistics, and five comparison checks re-ran full-column type
inference (five date parses plus a numeric parse) independently — the
dominant cost on large files.infer_col_type() rejects non-matching date formats from a 100-value head
sample before scanning the full column (results are identical; only the
rejection path gets cheaper).compute_col_stats() builds one data frame per column instead of one per
statistic row, cutting allocations on wide files.csv_skip (parallel to fwf_skip):
read_dataset() now forwards skip = config$csv_skip %||% 0L to
readr::read_delim(). This lets a config supply an explicit col_names
list and drop the file's original header row — required for delivery
files whose header repeats column names (e.g. a header that repeats
Name/Amount per bundled record), where the real header is unusable and
must be replaced positionally. Defaults to 0L, so existing configs are
byte-for-byte unaffected.?dqcheckr package help now resolves (the "_PACKAGE" doc block no
longer carries @noRd).csv_skip value (skip exceeding the row
count yields a zero-row frame).expect_error().@importFrom declarations for template-only packages.read_pass_rate_trend() function from drift.R.CP-07 type-change comparison now uses a single guard so it only runs the
non-numeric rate comparison when both snapshots classify the column as
numeric — eliminates spurious WARNs on type-changed columns.read_dataset() now forwards the col_names config key to
readr::read_delim(), so headerless CSVs are read correctly.read_dataset() now forwards the quote_char config key to
readr::read_delim() as quote =.rlang::abort() calls (27 sites) now carry a typed, two-level class
hierarchy (c("<specific>", "dqcheckr_error")), letting callers catch
errors broadly or precisely.helper.R.compare_snapshots() and list_snapshots() functions for drift analysis.
Compares any two historical snapshots and optionally renders an HTML drift
report with per-column statistical drift, schema changes, and trend charts.resolve_col_type() function: returns the effective type for a column,
respecting per-column type overrides set in the column_types config key.QC-15 outlier detection check (check_outliers()). Configured via
max_z_score and/or iqr_fence_multiplier; skipped silently when neither is
set.check_key_uniqueness() (QC-12) now supports composite keys: set
key_columns to a character vector in the dataset YAML.check_min_row_count() (QC-14) gains max_row_count and max_file_size_mb
thresholds.col_threshold() and table_threshold() added as internal helpers;
column_rules per-column threshold overrides are now correctly stored in
column_snapshots.threshold (G-01).render_report() uses
quarto::quarto_render() and returns NULL with a warning when Quarto CLI
is not installed.compare_schema() (CP-02) split into three separate result objects:
CP-02a (new columns), CP-02b (dropped columns), CP-02c (type changes).compare_non_numeric_rate() (CP-07) now always emits a result for every
eligible column, including PASS for columns where the rate did not increase
(G-02).run_timestamp is now stored in ISO-8601 UTC format
(2026-01-01T12:00:00Z) rather than local time (RC-07).snapshots table gains three columns on first use: comparison_mode,
render_status, and type_changed_cols_vs_previous. Existing 0.1.x
databases are auto-migrated on the first 0.2.0 run.PRAGMA foreign_keys = ON).detect_files() uses filename as a secondary sort when two files share the
same modification time, making folder-mode ordering deterministic (RC-01).check_allowed_values() (QC-09), compare_new_values() (CP-05), and
compare_dropped_values() (CP-06) cap observed at 20 values with an
"... and N more" suffix (RC-04, RC-05).check_non_numeric() (QC-11) gains a warn_non_numeric_rate config key for
a separate WARN threshold (C-01).compare_missing_rate() (CP-03) gains a missing_rate_change_severity
config key (warn / fail) (B-07).compare_column_order() (CP-08) gains a column_order_severity config key
that overrides the format-based default (C-02).compute_col_stats() stores the numeric mean under the key
numeric_parseable_mean (renamed from numeric_mean) to clarify that
non-parseable values are excluded from the calculation (C-04).compare_snapshots() uses the full merged per-dataset config for drift
threshold comparisons, so *** markers match the original check run for
datasets with rule_overrides (G-05/G-06).compute_col_stats() unused qc_results parameter removed (D-01).flag_new_columns, flag_dropped_columns, flag_type_changes (in CP-02) and
flag_column_order_change (CP-08) are now honoured. Setting any flag to false
suppresses the corresponding check from the report. Schema changes are still
tracked in the SQLite snapshot regardless of flags.type_inference_threshold is now configurable per dataset via rule_overrides
in the dataset YAML (or default_rules in the global config). Previously fixed
at 90%, it now defaults to 90% if not set. Affects QC-06, QC-07, QC-08, QC-11,
CP-02, CP-04, CP-05, CP-06, and CP-07.Initial release.
dqcheckr.yml and per-dataset YAML files.