Sources API¶
Source content remains untrusted data throughout normalization. The default registry accepts local files only and uses typed exceptions for format, security, resource, dependency, conversion, and timeout failures.
See Local Sources for supported formats, default limits, optional dependencies, and operational controls.
Registry methods¶
| Method | Return | Behavior |
|---|---|---|
convert(source, limits=...) |
SourceDocument |
Validate, snapshot, hash, and normalize one local source |
convert_for_svis(source, extraction_date=..., limits=...) |
SourceSvisConversionResult |
Add native XLSForm SVIS and routing semantics when available |
convert_with_native(source, svis, limits=...) |
SourceConversionResult |
Add an exact routing binding and optional native semantics |
All methods accept a local path or confined SourceBundle. They raise typed
SourceError subclasses for input, format, security, dependency, conversion,
timeout, and resource-limit failures. They do not accept bytes, streams, URLs, or
network paths.
Core source types¶
base ¶
Typed source port, normalized models, limits, and local-path confinement.
SourceLimits
dataclass
¶
SourceLimits(
max_source_bytes: int = 250 * 1024 * 1024,
max_pages: int = 2000,
max_archive_expanded_bytes: int = 1024 * 1024 * 1024,
max_archive_ratio: float = 100.0,
max_archive_entries: int = 10000,
max_archive_filename_chars: int = 512,
max_archive_path_depth: int = 20,
max_xml_part_bytes: int = 64 * 1024 * 1024,
max_xml_elements: int = 2000000,
max_xml_depth: int = 256,
max_cells: int = 2000000,
max_companions: int = 100,
deadline_seconds: float = 30 * 60.0,
)
SourceBundle
dataclass
¶
SourceBundle(
root: Path,
primary: Path,
companions: tuple[Path, ...] = (),
)
ResolvedSource
dataclass
¶
ResolvedSource(
root: Path,
primary: Path,
companions: tuple[Path, ...] = (),
primary_sha256: str | None = None,
companion_sha256: tuple[str, ...] = (),
)
Validated absolute paths passed from the source port to adapters.
SourceCoverage ¶
Bases: BaseModel
Complete immutable accounting for physical source conversion units.
SourceProvenance ¶
Bases: BaseModel
Available physical origin for normalized source content.
model_config
class-attribute
instance-attribute
¶
model_config = ConfigDict(extra='forbid', frozen=True)
reject_boolean_pages
classmethod
¶
reject_boolean_pages(value: object) -> object
Reject Boolean page values before Pydantic converts them to integers.
validate_section_path
classmethod
¶
validate_section_path(
value: tuple[str, ...],
) -> tuple[str, ...]
Normalize and reject empty source section labels.
SourceTable ¶
Bases: BaseModel
A table preserved as ordered text cells without formula execution.
model_config
class-attribute
instance-attribute
¶
model_config = ConfigDict(extra='forbid', frozen=True)
SourceBlock ¶
Bases: BaseModel
One ordered text or complete-table unit from an untrusted source.
model_config
class-attribute
instance-attribute
¶
model_config = ConfigDict(extra='forbid', frozen=True)
SourceDocument ¶
Bases: BaseModel
Deterministic normalized content; all source text remains untrusted data.
SourceDiagnostic ¶
Bases: BaseModel
Stable immutable diagnostic for one source or physical source unit.
model_config
class-attribute
instance-attribute
¶
model_config = ConfigDict(extra='forbid', frozen=True)
SourceError ¶
SourceError(message: str)
SourceInputError ¶
SourceInputError(message: str)
SourceFormatError ¶
SourceFormatError(message: str)
SourceSecurityError ¶
SourceSecurityError(message: str)
SourceDependencyError ¶
SourceDependencyError(message: str)
SourceConversionError ¶
SourceConversionError(message: str)
SourceTimeoutError ¶
SourceTimeoutError(message: str)
SourceLimitError ¶
SourceLimitError(limit: str, message: str)
Bases: SourceError
A configured source resource ceiling was exceeded.
SourceAdapter ¶
Bases: Protocol
Minimal adapter port used by the source registry.
convert ¶
convert(
source: ResolvedSource, *, limits: SourceLimits
) -> SourceDocument
Normalize one validated local source without external side effects.
resolve_local_source ¶
resolve_local_source(
source: LocalSource | SourceBundle,
*,
limits: SourceLimits = DEFAULT_SOURCE_LIMITS,
) -> ResolvedSource
Validate local-only input and confine bundle paths after resolution.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
source
|
LocalSource | SourceBundle
|
Existing local file or a bundle rooted in one local directory. |
required |
limits
|
SourceLimits
|
Resource ceilings applied to the primary and companion files. |
DEFAULT_SOURCE_LIMITS
|
Returns:
| Type | Description |
|---|---|
ResolvedSource
|
Absolute validated paths for the selected source. |
Raises:
| Type | Description |
|---|---|
SourceInputError
|
A path is remote, missing, not a file, or escapes its bundle root. |
SourceLimitError
|
File size or companion count exceeds a configured limit. |
snapshot_resolved_source ¶
snapshot_resolved_source(
source: ResolvedSource,
*,
limits: SourceLimits = DEFAULT_SOURCE_LIMITS,
) -> Iterator[ResolvedSource]
Yield a bounded private copy whose digest matches converted bytes.
validate_source_argument ¶
validate_source_argument(
source: object,
) -> LocalSource | SourceBundle
Validate the public local-source argument without accessing the filesystem.
Registry¶
registry ¶
Deterministic Tier 1 source adapter registry.
SourceRegistry ¶
SourceRegistry(adapters: Mapping[str, SourceAdapter])
Map verified file suffixes to minimal source adapters.
Create a suffix-to-adapter registry.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
adapters
|
Mapping[str, SourceAdapter]
|
Adapter mapping keyed by file suffix. Suffix matching is case-insensitive. |
required |
default
classmethod
¶
default() -> SourceRegistry
Create the verified Tier 1 registry without importing optional SDKs.
Returns:
| Type | Description |
|---|---|
SourceRegistry
|
A registry for PDF, DOCX, XLSX, CSV, HTML, Markdown, and text files. |
convert ¶
convert(
source: LocalSource | SourceBundle,
*,
limits: SourceLimits = DEFAULT_SOURCE_LIMITS,
) -> SourceDocument
Resolve one local source, verify its format, and normalize it.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
source
|
LocalSource | SourceBundle
|
Existing local file or confined source bundle. |
required |
limits
|
SourceLimits
|
Resource ceilings for validation and conversion. |
DEFAULT_SOURCE_LIMITS
|
Returns:
| Type | Description |
|---|---|
SourceDocument
|
Deterministic ordered blocks with physical source provenance. |
Raises:
| Type | Description |
|---|---|
SourceInputError
|
The local source path is invalid. |
SourceFormatError
|
The suffix is unsupported or conflicts with the file signature. |
SourceSecurityError
|
Untrusted content contains a prohibited vector. |
SourceDependencyError
|
An optional adapter dependency is unavailable. |
SourceLimitError
|
A configured resource ceiling is exceeded. |
SourceConversionError
|
The selected adapter cannot normalize the file. |
SourceTimeoutError
|
Killable conversion exceeds its deadline. |
convert_with_native ¶
convert_with_native(
source: LocalSource | SourceBundle,
svis: SurveySVIS,
*,
limits: SourceLimits = DEFAULT_SOURCE_LIMITS,
) -> SourceConversionResult
Normalize one source and add its exact binding and native semantics.
convert_for_svis ¶
convert_for_svis(
source: LocalSource | SourceBundle,
*,
extraction_date: date,
limits: SourceLimits = DEFAULT_SOURCE_LIMITS,
) -> SourceSvisConversionResult
Normalize a source and parse package-native SVIS when available.
SourceConversionResult
dataclass
¶
SourceConversionResult(
document: SourceDocument,
source_binding: RoutingSourceBinding,
native: NativeRoutingSemantics | None,
)
Additive normalized source result with binding and optional native routing.
SourceSvisConversionResult
dataclass
¶
SourceSvisConversionResult(
document: SourceDocument,
svis: SurveySVIS | None,
native: NativeRoutingSemantics | None,
)
Normalized source plus optional package-native SVIS and routing semantics.
Document adapters¶
docling ¶
Safe document adapters, including killable local Docling PDF conversion.
DoclingPdfAdapter ¶
DoclingPdfAdapter(
*,
converter: PdfConverter | None = None,
artifacts_path: str | PathLike[str] | None = None,
)
Normalize a local PDF in a fresh killable, network-disabled process.
DocxAdapter ¶
Parse DOCX XML as inert text after strict archive inspection.
HtmlAdapter ¶
Normalize visible HTML text and tables without loading active resources.
MarkdownAdapter ¶
Normalize UTF-8 Markdown while preserving preamble and complete tables.
TextAdapter ¶
Normalize local UTF-8 text without interpreting its instructions.
Tabular adapters¶
tabular ¶
Deterministic CSV and lazy-openpyxl XLSX source adapters.
CsvAdapter ¶
Normalize UTF-8 CSV rows with physical one-based row provenance.
convert_until ¶
convert_until(
source: ResolvedSource,
*,
limits: SourceLimits,
deadline: float,
) -> SourceDocument
Normalize CSV content under a caller-owned end-to-end deadline.
XlsxAdapter ¶
XlsxAdapter(
*, workbook_loader: WorkbookLoader | None = None
)
Normalize XLSX cells without formulas, macros, links, or eager workbook loading.
convert_until ¶
convert_until(
source: ResolvedSource,
*,
limits: SourceLimits,
deadline: float,
) -> SourceDocument
Normalize workbook content under a caller-owned end-to-end deadline.
XLSForm adapter¶
The default registry uses XlsFormAdapter for .xlsx. It keeps ordinary XLSX
normalization and adds native SVIS, choices, relevance, group, repeat, and source
binding data when the workbook has an XLSForm survey sheet.
xlsform ¶
Bounded core XLSForm adapter with versioned native-routing support.
XLSFORM_SUPPORT_MATRIX
module-attribute
¶
XLSFORM_SUPPORT_MATRIX: Mapping[str, str] = (
MappingProxyType(
{
"tested_profile": "XLSForm 1.3 core; ODK-compatible workbook structure",
"question_types": "acknowledge,audio,barcode,calculate,date,dateTime,decimal,end,file,geopoint,geoshape,geotrace,hidden,image,integer,note,range,select_multiple,select_multiple_from_file,select_one,select_one_from_file,start,text,time,today,username,video",
"diagnostic_codes": "METADATA_INCOMPLETE,XLSFORM_CHOICE_LIST_MISSING,XLSFORM_FEATURE_UNSUPPORTED,XLSFORM_FUNCTION_UNSUPPORTED,XLSFORM_EXPRESSION_UNSUPPORTED,XLSFORM_REFERENCE_UNRESOLVED,XLSFORM_TYPE_UNSUPPORTED",
"survey_choices_settings": "preserved",
"multilingual_labels": "preserved",
"groups": "logical_containment",
"repeats": "logical_template",
"reference_comparisons": "exact",
"selected": "exact",
"boolean_and_or_not": "exact",
"functions_other_than_selected": "opaque",
"arithmetic": "opaque",
"constraints": "preserved_not_flow",
"calculations": "preserved_not_flow",
"choice_filters": "preserved_not_flow",
"excel_formulas_macros_external_links": "rejected",
"external_choices": "confined_bundle_only",
"limitations": "no formula, macro, external-link, dynamic-instance, rank, or XML-external evaluation; non-projected functions remain exact source text",
}
)
)
XlsFormAdapter ¶
XlsFormAdapter(*, xlsx_adapter: XlsxAdapter | None = None)
Preserve XLSX normalization and add native semantics for XLSForm workbooks.
convert ¶
convert(
source: ResolvedSource, *, limits: SourceLimits
) -> SourceDocument
Return the unchanged normalized XLSX document contract.
convert_until ¶
convert_until(
source: ResolvedSource,
*,
limits: SourceLimits,
deadline: float,
) -> SourceDocument
Normalize the workbook under one source-owned deadline.
convert_native ¶
convert_native(
source: ResolvedSource,
document: SourceDocument,
*,
limits: SourceLimits,
) -> NativeRoutingSemantics | None
Parse an XLSForm when a case-insensitive survey sheet is present.
convert_native_until ¶
convert_native_until(
source: ResolvedSource,
document: SourceDocument,
*,
limits: SourceLimits,
deadline: float,
) -> NativeRoutingSemantics | None
Parse native XLSForm semantics under the active source deadline.
convert_svis_until ¶
convert_svis_until(
source: ResolvedSource,
document: SourceDocument,
*,
limits: SourceLimits,
deadline: float,
extraction_date: date,
) -> tuple[SurveySVIS, NativeRoutingSemantics] | None
Build authoritative SVIS and routing semantics from one XLSForm parse.
Chunking¶
chunking ¶
Deterministic token-aware chunking with table and overlap provenance.
TokenEstimator ¶
Bases: Protocol
Minimal tokenizer-independent estimation port.
ConservativeTokenEstimator ¶
Dependency-free fallback that budgets one token per UTF-8 byte.
RepeatedRowOrigin ¶
Bases: BaseModel
Physical table and one-based source row for repeated content.
model_config
class-attribute
instance-attribute
¶
model_config = ConfigDict(extra='forbid', frozen=True)
RepeatedRow ¶
Bases: BaseModel
Inventory entry for an exact normalized row occurring more than once.
model_config
class-attribute
instance-attribute
¶
model_config = ConfigDict(extra='forbid', frozen=True)
SourceChunk ¶
Bases: BaseModel
One stable source-order chunk with explicit overlap ownership.
model_config
class-attribute
instance-attribute
¶
model_config = ConfigDict(extra='forbid', frozen=True)
ChunkedDocument ¶
Bases: BaseModel
Deterministic chunks and completeness inventory for one source.
model_config
class-attribute
instance-attribute
¶
model_config = ConfigDict(extra='forbid', frozen=True)
chunk_document ¶
chunk_document(
document: SourceDocument,
*,
max_tokens: int,
overlap_tokens: int = 0,
estimator: TokenEstimator | None = None,
) -> ChunkedDocument
Chunk without dropping preamble, complete tables, or source order.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
document
|
SourceDocument
|
Normalized source document to chunk. |
required |
max_tokens
|
int
|
Hard maximum for every final chunk, including overlap. |
required |
overlap_tokens
|
int
|
Target budget for complete prior text blocks copied into the next chunk. |
0
|
estimator
|
TokenEstimator | None
|
Token estimator. The conservative dependency-free estimator is used when omitted. |
None
|
Returns:
| Type | Description |
|---|---|
ChunkedDocument
|
Stable chunks plus an inventory of exact repeated table rows. |
Raises:
| Type | Description |
|---|---|
ValueError
|
|
OCR cache validation¶
ocr ¶
Offline checksum validation for approved local EasyOCR artifacts.
OcrArtifact
dataclass
¶
OcrArtifact(filename: str, size: int, sha256: str)
OcrArtifactValidation
dataclass
¶
OcrArtifactValidation(
artifact: OcrArtifact,
status: OcrValidationStatus,
checksum_checked: bool,
)
Accurate local validation state for one approved OCR archive.
validate_ocr_cache ¶
validate_ocr_cache(
cache: Path,
*,
manifest: Sequence[
OcrArtifact
] = APPROVED_OCR_ARTIFACTS,
) -> tuple[OcrArtifactValidation, ...]
Validate local files without fetching or redistributing artifacts.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
cache
|
Path
|
Local directory that contains OCR archives. |
required |
manifest
|
Sequence[OcrArtifact]
|
Expected artifact names, byte sizes, and SHA-256 digests. |
APPROVED_OCR_ARTIFACTS
|
Returns:
| Type | Description |
|---|---|
tuple[OcrArtifactValidation, ...]
|
One validation result per manifest item in manifest order. |
Note
Validation is caller-enforced. PDF conversion does not call this function automatically.
main ¶
main(
*,
environ: Mapping[str, str] | None = None,
stdout: TextIO | None = None,
stderr: TextIO | None = None,
) -> int
Validate the configured local OCR cache and return a deterministic exit code.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
environ
|
Mapping[str, str] | None
|
Environment mapping used to locate the cache. |
None
|
stdout
|
TextIO | None
|
Stream for per-artifact status and successful summary output. |
None
|
stderr
|
TextIO | None
|
Stream for configuration and validation failures. |
None
|
Returns:
| Type | Description |
|---|---|
int
|
|
int
|
|