{ "manifest_generated_at": "2026-10-03T15:11:39Z", "schema_version_format": "integer-monotonic", "sha256_target": "gzipped_artifact", "current_week": "2026-W40", "artifacts": [ { "filename": "990data_public.db.gz", "key": "weekly/2026-W40/990data_public.db.gz", "url": "https://dumps.datadawn.org/weekly/2026-W40/990data_public.db.gz", "size_bytes": 6883421078, "sha256_gzipped": "ce8cf78f006d276bdd8c3537d74e918e350afcdae03a0ebd40f83b552b90d291", "generated_at": "2026-10-03T15:00:50Z", "schema_version": 1 }, { "filename": "aphis.db.gz", "key": "weekly/2026-W40/aphis.db.gz", "url": "https://dumps.datadawn.org/weekly/2026-W40/aphis.db.gz", "size_bytes": 20063482, "sha256_gzipped": "f6a344b59f4aca430fe926b9f9106329bd3832aa9e6d260c0db9509eb2045785", "generated_at": "2026-10-03T15:00:52Z", "schema_version": 1 }, { "filename": "fara.db.gz", "key": "weekly/2026-W40/fara.db.gz", "url": "https://dumps.datadawn.org/weekly/2026-W40/fara.db.gz", "size_bytes": 10124878, "sha256_gzipped": "5ebc0f9659fd3b61cf054617ba9fc787385a7b8eb02ebb9ca3124b20909f71b8", "generated_at": "2026-10-03T15:00:53Z", "schema_version": 1 }, { "filename": "lobbying.db.gz", "key": "weekly/2026-W40/lobbying.db.gz", "url": "https://dumps.datadawn.org/weekly/2026-W40/lobbying.db.gz", "size_bytes": 2119787417, "sha256_gzipped": "947d88cdb02951c84a6aa82fc9f2adde3d393aed3eefeb6a5368e19c5fb19297", "generated_at": "2026-10-03T15:02:19Z", "schema_version": 1 }, { "filename": "open_comments.db.gz", "key": "weekly/2026-W40/open_comments.db.gz", "url": "https://dumps.datadawn.org/weekly/2026-W40/open_comments.db.gz", "size_bytes": 132851, "sha256_gzipped": "58f874b8001e4fed4dac838efab080b1498db664b20672742f42aa1142c7ef34", "generated_at": "2026-10-03T15:02:19Z", "schema_version": 1, "grain_note": "documents.comment_end_date is MID-CORRECTION (DataDawn queue #498). Before the correction it holds the UTC-rolled calendar day AFTER the official 11:59:59 PM Eastern deadline -- one day LATE on every end-of-day close. After it, it holds the Eastern calendar date. No field marks which one this copy has. SELF-TEST: pick any row with both columns set and compare substr(comment_end_datetime,1,10) to comment_end_date -- EQUAL means the OLD (one-day-late) grain; comment_end_date one day EARLIER means the corrected grain. documents.comment_end_datetime is the full UTC instant and is AUTHORITATIVE in every dump ever published, before and after the correction -- prefer it unconditionally and convert to America/New_York. Do not subtract a fixed offset: 36% of the corpus is EST-era and that reintroduces the same one-day error." }, { "filename": "openregs.db.gz", "key": "weekly/2026-W40/openregs.db.gz", "url": "https://dumps.datadawn.org/weekly/2026-W40/openregs.db.gz", "size_bytes": 21968512624, "sha256_gzipped": "2c3574e80099fedfe4c6b1f246e786fc30c90df576dbd8e96b7db8d4e5d85cd1", "generated_at": "2026-10-03T15:11:39Z", "schema_version": 1, "grain_note": "READ THIS FIRST -- which kind of copy is this? Run SELECT value FROM build_metadata WHERE key='bulk_merge'. 'disabled' means this copy has NO bulk-export text at all: the comment_texts and comment_bodies tables are ABSENT ENTIRELY, and the only comment text present is API text in comment_details.comment_text (a small minority of comments). If you got 'disabled', the rest of the bulk paragraph below does not describe your copy. A value starting 'merged:' means bulk text IS present and everything below applies; the digits after the colon are the comment row count that merge produced. ON A 'merged:' COPY, comment text comes from TWO regulations.gov channels and they are not interchangeable. API text lives in comment_details.comment_text. Bulk-export text lives in comment_texts.comment_text, reached from a comment id through comment_bodies.body_sha256. Which table a row's text came from IS its provenance -- there is no stored source column. For the bulk channel: \"Bulk export text has been decoded and had HTML markup removed; wording is otherwise as published.\" API text is NOT normalised and still carries the publisher's markup, so the same comment can differ byte-for-byte between the two channels while saying the same thing. Do not diff one against the other expecting equality. TEXT IS WITHHELD -- deliberately, and is not missing data -- for rows marked withdrawn and for rows marked \"No longer posted on regulations.gov\". Such a row is kept with its metadata and carries no text in either channel present in this copy. An empty comment text on one of those rows is the intended state. A THIRD KIND OF WITHHELD ROW carries comments.suppression_reason of the form operator::: DataDawn, not the source, withheld the text. The served page says, for class privacy: \"Text withheld by DataDawn on (privacy: personal information of a private individual beyond what the source redacted). The comment itself remains on the public docket at regulations.gov. How to request a correction: https://datadawn.org/methodology#independence\"; for class legal: \"Text withheld by DataDawn on in response to a legal request. The comment itself remains on the public docket at regulations.gov. How to request a correction: https://datadawn.org/methodology#independence\"; for class review: \"Text temporarily withheld by DataDawn on while a report about it is reviewed. The comment itself remains on the public docket at regulations.gov. How to request a correction: https://datadawn.org/methodology#independence\" SELF-TEST 2, was withheld-row suppression applied: SELECT value FROM build_metadata WHERE key='withdrawn_text_suppressed' -- returns a JSON object (phase, counts, ruling) when it ran and NO ROW AT ALL when it did not. A copy with no such row predates the suppression and may still carry text on withdrawn rows. SELF-TEST 3, which withheld classes exist in this copy and how to select them: comments.suppression_reason is the marker. It is NULL on a row whose text was not withheld, 'withdrawn' on the withdrawn class, and 'not_posted' on the \"No longer posted on regulations.gov\" class. SELECT suppression_reason, COUNT(*) FROM comments GROUP BY 1 -- gives the whole picture. If your copy's comments table has NO suppression_reason column it predates this marker: only the withdrawn class exists in it, and comments.withdrawn = 1 identifies that class. DO NOT USE comments.withdrawn = 1 AS THE WITHHELD TEST ON A COPY THAT HAS THE MARKER. The two are not the same set and are not meant to be. comments.withdrawn reports what the source flags; suppression_reason reports what this copy withheld. A small number of rows are withheld while carrying withdrawn = 0 -- those are rows where the bulk export's withdrawal REASON TEXT and its withdrawal BOOLEAN disagree, which we resolve toward withholding unless a dated per-id check against regulations.gov confirms the comment is live there. Selecting on withdrawn alone therefore UNDERCOUNTS the withheld set." } ] }