Skip to content

Processing

processing

Processing a feature collection by rules that are kept as data.

Each subpackage applies one rule file from xmas_core/resources to a feature collection: transform migrates it between XPlanung versions, and style derives stylesheetId and schriftinhalt for its presentational objects. common reads and writes those files.

Nothing is imported here, so that reaching for one subpackage does not load the other's rules.

common

Reading and writing the rule files that transform and style are driven by.

Both rule files are YAML written by hand, and both are read the same way: through load_yaml, which refuses what PyYAML would otherwise read silently as something other than what was written.

The rest of this module is the part of scripts/normalize_migration_rules.py and scripts/normalize_style_rules.py that does not depend on the document. A normalizer validates the file into its model, has its own write turn the model back into YAML, and keeps the result only if it validates to the same thing. It writes that result through dump, so both files look alike:

  • The comment block at the top of the file is kept as it is. A YAML comment anywhere below it is refused, since it would not survive being written back; notes are comment: fields.
  • Lists are indented under their key, strings are double-quoted where they need quoting, and an object shared by several places is anchored once under a name chosen by the caller.

Comment

Bases: str

A note, written folded when it does not fit on its key's line.

NormalizeError

Bases: Exception

The file cannot be normalized as it is.

RuleFileError

Bases: ValueError

A rule file does not describe the rules it claims to.

UniqueKeyLoader

Bases: SafeLoader

A loader that refuses a duplicate mapping key.

PyYAML keeps the last of a repeated key, which is the same silent shadowing the method dispatch had: the 4.1 rules defined four feature types twice each and only the later definition ever ran.

construct_mapping(node, deep=False)

Build the mapping, raising on any key that appears twice, or a merge key.

A merge key (<<: *template) is refused rather than resolved: a template is a whole operation, rule or clause, used through a plain alias, so an operation always states every field it has where it is written.

Source code in xmas_core/processing/common.py
def construct_mapping(
    self, node: yaml.MappingNode, deep: bool = False
) -> dict[Hashable, Any]:
    """Build the mapping, raising on any key that appears twice, or a merge key.

    A merge key (`<<: *template`) is refused rather than resolved: a template is a whole
    operation, rule or clause, used through a plain alias, so an operation always states
    every field it has where it is written.
    """
    seen = set()
    for key_node, _ in node.value:
        if key_node.tag == "tag:yaml.org,2002:merge":
            raise RuleFileError(
                f"line {key_node.start_mark.line + 1}: merge keys (<<) are not used; "
                "write the fields out"
            )
        key = self.construct_object(key_node, deep=deep)
        if isinstance(key, bool):
            # YAML 1.1 reads on/off/yes/no as booleans, so an unquoted one silently
            # stops being the key that was written
            raise RuleFileError(
                f"{key_node.value!r} at line {key_node.start_mark.line + 1} is a YAML "
                "boolean, not a key; quote it or use a different name"
            )
        if key in seen:
            raise RuleFileError(
                f"duplicate key {key!r} at line {key_node.start_mark.line + 1}"
            )
        seen.add(key)
    return super().construct_mapping(node, deep)

dump(node, names=None)

Write node in the rule files' style.

Parameters:

Name Type Description Default
node Any

Plain data: dicts, lists, strings, numbers and Comments.

required
names dict[int, str] | None

Anchor names by id() of the object to anchor. An object that appears more than once is anchored where it is first written and aliased everywhere else; one not named here gets PyYAML's id001.

None

Returns:

Type Description
str

The YAML text.

Source code in xmas_core/processing/common.py
def dump(node: Any, names: dict[int, str] | None = None) -> str:
    """Write `node` in the rule files' style.

    Args:
        node: Plain data: dicts, lists, strings, numbers and `Comment`s.
        names: Anchor names by `id()` of the object to anchor. An object that appears more
            than once is anchored where it is first written and aliased everywhere else;
            one not named here gets PyYAML's `id001`.

    Returns:
        The YAML text.
    """
    anchors = names or {}

    class Dumper(yaml.SafeDumper):
        def __init__(self, *args: Any, **kwargs: Any) -> None:
            super().__init__(*args, **kwargs)
            self.anchor_names: dict[Any, str] = {}

        def increase_indent(self, flow: bool = False, indentless: bool = False) -> None:
            # a list sits two spaces under its key, not level with it
            return super().increase_indent(flow, False)

        def represent_data(self, data: Any) -> Any:
            node = super().represent_data(data)
            if id(data) in anchors:
                self.anchor_names[node] = anchors[id(data)]
            return node

        def generate_anchor(self, node: Any) -> Any:
            return self.anchor_names.get(node) or super().generate_anchor(node)

        def choose_scalar_style(self) -> Any:
            style = super().choose_scalar_style()
            return '"' if style == "'" else style

        def write_folded(self, text: str) -> None:
            width, self.best_width = self.best_width, COMMENT_WIDTH
            try:
                super().write_folded(text)
            finally:
                self.best_width = width

    # a note that fits on its key's line stays there; anything longer is folded under it
    Dumper.add_representer(
        Comment,
        lambda d, v: d.represent_scalar(
            "tag:yaml.org,2002:str",
            str(v),
            style=">" if "\n" in v or len(v) > COMMENT_WIDTH - 30 else None,
        ),
    )
    return yaml.dump(
        node, Dumper=Dumper, sort_keys=False, allow_unicode=True, width=4096
    )

fold_comments(node)

Mark every comment value below node as a Comment, in place.

In place, so an object shared by several places stays one object and keeps its anchor.

Source code in xmas_core/processing/common.py
def fold_comments(node: Any) -> None:
    """Mark every `comment` value below `node` as a `Comment`, in place.

    In place, so an object shared by several places stays one object and keeps its anchor.
    """
    seen: set[int] = set()
    stack = [node]
    while stack:
        current = stack.pop()
        if id(current) in seen:
            continue
        seen.add(id(current))
        if isinstance(current, dict):
            if isinstance(current.get("comment"), str):
                current["comment"] = Comment(current["comment"])
            stack.extend(current.values())
        elif isinstance(current, list):
            stack.extend(current)

load_yaml(text)

Read a rule file's YAML.

Parameters:

Name Type Description Default
text str

The file's contents.

required

Raises:

Type Description
RuleFileError

A key is written twice, is a merge key or reads as a boolean.

YAMLError

text is not YAML.

Returns:

Type Description
Any

The document, with every alias resolved to the object its anchor names.

Source code in xmas_core/processing/common.py
def load_yaml(text: str) -> Any:
    """Read a rule file's YAML.

    Args:
        text: The file's contents.

    Raises:
        RuleFileError: A key is written twice, is a merge key or reads as a boolean.
        yaml.YAMLError: `text` is not YAML.

    Returns:
        The document, with every alias resolved to the object its anchor names.
    """
    # B506 matches the name `Loader=` is given, not what it derives from: UniqueKeyLoader
    # is a SafeLoader, and yaml.safe_load is exactly this call with the plain one
    return yaml.load(text, Loader=UniqueKeyLoader)  # nosec B506

normalize_document(text, *, validate, write, same=operator.eq)

Return text in canonical form, having checked it still says the same thing.

Parameters:

Name Type Description Default
text str

The rule file's contents.

required
validate Callable[[Any], D]

Builds the document's model from the loaded YAML.

required
write Callable[[Any, D], str]

Writes the file below its header, given the loaded YAML and the model.

required
same Callable[[D, D], bool]

Whether two models say the same thing. Equality unless the model holds something that normalizing is allowed to change.

eq

Raises:

Type Description
NormalizeError

The file is not valid, holds a YAML comment below its header, or would no longer say the same thing once written.

Returns:

Type Description
str

The canonical text, the header comment block included.

Source code in xmas_core/processing/common.py
def normalize_document[D](
    text: str,
    *,
    validate: Callable[[Any], D],
    write: Callable[[Any, D], str],
    same: Callable[[D, D], bool] = operator.eq,
) -> str:
    """Return `text` in canonical form, having checked it still says the same thing.

    Args:
        text: The rule file's contents.
        validate: Builds the document's model from the loaded YAML.
        write: Writes the file below its header, given the loaded YAML and the model.
        same: Whether two models say the same thing. Equality unless the model holds
            something that normalizing is allowed to change.

    Raises:
        NormalizeError: The file is not valid, holds a YAML comment below its header, or
            would no longer say the same thing once written.

    Returns:
        The canonical text, the header comment block included.
    """
    lines = text.splitlines()
    header = _header(lines)
    _refuse_yaml_comments(lines, len(header))
    raw = _load(text)
    document = _validate(validate, raw)
    head = "\n".join(line for line in header if line.strip())
    result = (head + "\n\n" if head else "") + write(raw, document)
    if not same(_validate(validate, _load(result)), document):
        raise NormalizeError(
            "normalizing would change what the rules say; nothing was written"
        )
    return result

run_normalizer(argv, *, normalize, default, description)

Normalize the given files in place, or report with --check.

Parameters:

Name Type Description Default
argv Sequence[str] | None

The command line, without the program name; None reads sys.argv.

required
normalize Callable[[str], str]

Turns a file's text into its canonical form.

required
default Path

The file to normalize when none is given.

required
description str

The first line of the script's help.

required

Returns:

Type Description
int

0 when nothing changed, 1 when a file was (or with --check, would be) rewritten,

int

2 on an error.

Source code in xmas_core/processing/common.py
def run_normalizer(
    argv: Sequence[str] | None,
    *,
    normalize: Callable[[str], str],
    default: Path,
    description: str,
) -> int:
    """Normalize the given files in place, or report with `--check`.

    Args:
        argv: The command line, without the program name; None reads `sys.argv`.
        normalize: Turns a file's text into its canonical form.
        default: The file to normalize when none is given.
        description: The first line of the script's help.

    Returns:
        0 when nothing changed, 1 when a file was (or with `--check`, would be) rewritten,
        2 on an error.
    """
    parser = argparse.ArgumentParser(description=description)
    parser.add_argument("--check", action="store_true", help="report, do not write")
    parser.add_argument("files", nargs="*", type=Path, default=[default])
    args = parser.parse_args(argv)
    status = 0
    for path in args.files:
        text = path.read_text("utf-8")
        try:
            result = normalize(text)
        except NormalizeError as e:
            print(f"{path}: {e}", file=sys.stderr)
            return 2
        if result == text:
            continue
        status = 1
        if args.check:
            sys.stdout.writelines(
                difflib.unified_diff(
                    text.splitlines(keepends=True),
                    result.splitlines(keepends=True),
                    str(path),
                    str(path),
                )
            )
        else:
            path.write_text(result, "utf-8")
            print(f"normalized {path}")
    return status

transform

Migrating a feature collection from one XPlanung version to the next.

result = migrate(collection, "6.1")
result.collection            # the migrated collection
result.report.by_feature()   # what changed, keyed by feature id

migrate walks the whole path - 4.1 to 6.1 is three hops - and one report covers all of it. Every hop's rules live in one file, resources/migration/rules.yaml, as one entry of its migrations: list.

MigrationReport

Bases: BaseModel

Every issue raised while migrating a collection, across all hops.

errors property

The issues that failed the migration.

ok property

Whether the migration produced no errors.

warnings property

The issues the migration continued past.

add(severity, code, message, **fields)

Append an issue. Unknown keyword arguments land in meta.

Source code in xmas_core/processing/transform/report.py
def add(
    self,
    severity: Literal["error", "warning"],
    code: ErrorCode | WarnCode,
    message: str,
    **fields: Any,
) -> Issue:
    """Append an issue. Unknown keyword arguments land in `meta`."""
    known = {
        k: fields.pop(k)
        for k in ("feature_id", "feature_type", "property", "step")
        if k in fields
    }
    issue = Issue(
        severity=severity, code=code, message=message, meta=fields, **known
    )
    self.issues.append(issue)
    return issue

by_feature()

Group the issues by the feature they belong to, in the order they were raised.

Issues with no feature - a collection-level failure - are not in the result.

Source code in xmas_core/processing/transform/report.py
def by_feature(self) -> dict[UUID, list[Issue]]:
    """Group the issues by the feature they belong to, in the order they were raised.

    Issues with no feature - a collection-level failure - are not in the result.
    """
    grouped: dict[UUID, list[Issue]] = defaultdict(list)
    for issue in self.issues:
        if issue.feature_id is not None:
            grouped[issue.feature_id].append(issue)
    return dict(grouped)

error(code, message, **fields)

Record an error. An error aborts the migration once the current phase ends.

Source code in xmas_core/processing/transform/report.py
def error(self, code: ErrorCode, message: str, **fields: Any) -> Issue:
    """Record an error. An error aborts the migration once the current phase ends."""
    return self.add("error", code, message, **fields)

warning(code, message, **fields)

Record a warning. The migration continues.

Source code in xmas_core/processing/transform/report.py
def warning(self, code: WarnCode, message: str, **fields: Any) -> Issue:
    """Record a warning. The migration continues."""
    return self.add("warning", code, message, **fields)

MigrationResult(collection, report) dataclass

What migrate returns: the collection and the report.

A dataclass rather than a model, because the collection is the payload and nothing should be tempted to serialise the two together - report dumps on its own.

migrate(collection, to_version, *, report=None)

Migrate a collection to to_version, however many hops that takes.

One report covers the whole run: every issue records the hop that raised it, so a 4.1 -> 6.1 migration still says which step dropped an attribute.

Parameters:

Name Type Description Default
collection FeatureCollection

The collection to migrate.

required
to_version str

The wanted XPlanung version.

required
report MigrationReport | None

An existing report to collect into, if the caller is already holding one.

None

Raises:

Type Description
ValueError

to_version is not reachable from the collection's version.

MigrationError

A hop failed; the report says which features and why.

UnsupportedFeatureError

The collection holds a feature with no mapping at all.

Returns:

Type Description
MigrationResult

The migrated collection and the report.

Source code in xmas_core/processing/transform/__init__.py
def migrate(
    collection: FeatureCollection,
    to_version: str,
    *,
    report: MigrationReport | None = None,
) -> MigrationResult:
    """Migrate a collection to `to_version`, however many hops that takes.

    One report covers the whole run: every issue records the hop that raised it, so a
    4.1 -> 6.1 migration still says which step dropped an attribute.

    Args:
        collection: The collection to migrate.
        to_version: The wanted XPlanung version.
        report: An existing report to collect into, if the caller is already holding one.

    Raises:
        ValueError: `to_version` is not reachable from the collection's version.
        MigrationError: A hop failed; the report says which features and why.
        UnsupportedFeatureError: The collection holds a feature with no mapping at all.

    Returns:
        The migrated collection and the report.
    """
    from xmas_core.processing.transform.engine import run_hop
    from xmas_core.processing.transform.rules import load

    report = report if report is not None else MigrationReport()
    version = collection.appschema.version
    for step in migration_path(version, to_version):
        logger.info(f"Migrating from version {version} to {step}")
        collection = run_hop(collection, load(version, step), report)
        version = step
    return MigrationResult(collection=collection, report=report)

migration_path(from_version, to_version)

The versions to migrate through to get from from_version to to_version.

Parameters:

Name Type Description Default
from_version str

The collection's current version.

required
to_version str

The wanted version.

required

Raises:

Type Description
ValueError

to_version is not reachable from from_version.

Returns:

Type Description
list[str]

The intermediate targets, in order, excluding from_version. Empty if the

list[str]

collection is already at to_version.

Source code in xmas_core/processing/transform/__init__.py
def migration_path(from_version: str, to_version: str) -> list[str]:
    """The versions to migrate through to get from `from_version` to `to_version`.

    Args:
        from_version: The collection's current version.
        to_version: The wanted version.

    Raises:
        ValueError: `to_version` is not reachable from `from_version`.

    Returns:
        The intermediate targets, in order, excluding `from_version`. Empty if the
        collection is already at `to_version`.
    """
    path: list[str] = []
    current = from_version
    while current != to_version:
        if not (nxt := MIGRATIONS.get(current)):
            raise ValueError(
                f"Migration from version {from_version} to {to_version} not yet implemented"
            )
        path.append(nxt)
        current = nxt
    return path

report

The report a migration produces, and the error raised when one fails.

A migration is expected to change data in ways a downstream application has to look at: an attribute the target version no longer has, a value with no equivalent code, a whole feature the schema dropped. Every such finding is an Issue carrying the feature_id it belongs to, so the consumer can put it next to the feature rather than parse it out of a log line:

result = migrate(collection, "6.1")
for feature_id, issues in result.report.by_feature().items():
    ...

One report covers the whole run. A 4.1 -> 6.1 migration is three hops, and each issue records the step that raised it, so the report stays legible after the collection has been through all of them.

Issue

Bases: BaseModel

One finding about one feature.

code serialises to its name ("ATTRIBUTE_DROPPED"), the machine-readable token xmas_core.codes documents consumers to branch on, and validates back from it.

MigrationError(report)

Bases: BaseError

A migration failed. report holds every issue, including the errors that stopped it.

Carry the report of the run that failed.

Source code in xmas_core/processing/transform/report.py
def __init__(self, report: MigrationReport):
    """Carry the report of the run that failed."""
    self.report = report
    errors = report.errors
    super().__init__(
        f"Migration failed with {len(errors)} error(s), "
        "see `MigrationError.report` for details."
    )

MigrationReport

Bases: BaseModel

Every issue raised while migrating a collection, across all hops.

errors property

The issues that failed the migration.

ok property

Whether the migration produced no errors.

warnings property

The issues the migration continued past.

add(severity, code, message, **fields)

Append an issue. Unknown keyword arguments land in meta.

Source code in xmas_core/processing/transform/report.py
def add(
    self,
    severity: Literal["error", "warning"],
    code: ErrorCode | WarnCode,
    message: str,
    **fields: Any,
) -> Issue:
    """Append an issue. Unknown keyword arguments land in `meta`."""
    known = {
        k: fields.pop(k)
        for k in ("feature_id", "feature_type", "property", "step")
        if k in fields
    }
    issue = Issue(
        severity=severity, code=code, message=message, meta=fields, **known
    )
    self.issues.append(issue)
    return issue

by_feature()

Group the issues by the feature they belong to, in the order they were raised.

Issues with no feature - a collection-level failure - are not in the result.

Source code in xmas_core/processing/transform/report.py
def by_feature(self) -> dict[UUID, list[Issue]]:
    """Group the issues by the feature they belong to, in the order they were raised.

    Issues with no feature - a collection-level failure - are not in the result.
    """
    grouped: dict[UUID, list[Issue]] = defaultdict(list)
    for issue in self.issues:
        if issue.feature_id is not None:
            grouped[issue.feature_id].append(issue)
    return dict(grouped)

error(code, message, **fields)

Record an error. An error aborts the migration once the current phase ends.

Source code in xmas_core/processing/transform/report.py
def error(self, code: ErrorCode, message: str, **fields: Any) -> Issue:
    """Record an error. An error aborts the migration once the current phase ends."""
    return self.add("error", code, message, **fields)

warning(code, message, **fields)

Record a warning. The migration continues.

Source code in xmas_core/processing/transform/report.py
def warning(self, code: WarnCode, message: str, **fields: Any) -> Issue:
    """Record a warning. The migration continues."""
    return self.add("warning", code, message, **fields)

MigrationResult(collection, report) dataclass

What migrate returns: the collection and the report.

A dataclass rather than a model, because the collection is the payload and nothing should be tempted to serialise the two together - report dumps on its own.

ops

The two vocabularies a migration is written in.

A rule operation (single) works on one feature: it is handed the plain dict that feature was dumped to and nothing else, so it cannot reach another feature even by accident. A collection operation (collection) is the exception, and it is a separate module rather than a flag: pull and push are the only two, they run in their own phase, and keeping them apart is what makes the rule vocabulary safe to read.

Both are data. Every operation is a pydantic model built from resources/migration/rules.yaml, the vocabulary is closed, and there is no escape hatch back into Python.

RuleContext(report, step, migration_record=dict(), removed=set(), feature=None, target_type=None, plan_prefix=None) dataclass

What an operation is handed besides the feature dict.

One instance per hop. It deliberately does not hold the collection: a rule operates on one feature, and the phase that may look at others gets a context of its own.

feature = None class-attribute instance-attribute

The feature currently being migrated, as (id, type); set by the engine per feature.

A data type has no id of its own, so a rule that warns about one would otherwise produce an issue no consumer could attach to anything. This is what it falls back to.

plan_prefix = None class-attribute instance-attribute

The document's plan type (BP, FP, ...), computed once per hop by the engine.

A property of the collection rather than of any rule, which is why it is resolved here for the by_plan of set and retype rather than stated in the rule file.

target_type = None class-attribute instance-attribute

The feature type the current feature will be validated as; retype writes it.

It is the feature's identity rather than one of its values, so it lives here instead of in the dict the operations mutate - where it sat beside the real attributes, could be read by a guard as though the feature declared such a property, and had to be popped back out before the dict could be validated. A data type has no type of its own here, so this is None while a rule reaches into one.

record(obj, key, value)

Note what happened to an attribute, so the art-xpath remap can follow it.

key and value are (name, index, ...) tuples in the shape parse_art_xpath produces; a value of None means the attribute is gone, and a list of tuples that it is several entries now.

The first entry of a repeatable attribute is filed under both spellings. xplan:typ and xplan:typ[1] name the same entry and a presentation object may be written either way, but parse_art_xpath keeps them apart as (typ, None) and (typ, 0) while enrich_attr_tuple puts them back together on the way out - so a record answering to only one of them silently left the other pointing at an attribute the target class no longer has. That was 58 art strings across 20 documents of tests/data, every one of them a fold source named without an index.

Source code in xmas_core/processing/transform/ops/single.py
def record(self, obj: dict[str, Any], key: Any, value: Any) -> None:
    """Note what happened to an attribute, so the `art`-xpath remap can follow it.

    `key` and `value` are `(name, index, ...)` tuples in the shape
    [`parse_art_xpath`][xmas_core.processing.style.parse_art_xpath] produces; a `value` of None
    means the attribute is gone, and a list of tuples that it is several entries now.

    The first entry of a repeatable attribute is filed under **both** spellings.
    `xplan:typ` and `xplan:typ[1]` name the same entry and a presentation object may be
    written either way, but `parse_art_xpath` keeps them apart as `(typ, None)` and
    `(typ, 0)` while `enrich_attr_tuple` puts them back together on the way out - so a
    record answering to only one of them silently left the other pointing at an
    attribute the target class no longer has. That was 58 `art` strings across 20
    documents of `tests/data`, every one of them a `fold` source named without an index.
    """
    record = self.migration_record.setdefault(obj.get("id"), {})
    record[key] = value
    # ponytail: the first index pair only. A deeper step could be spelled either way too,
    # but no operation writes a record key that varies past the first pair; widen this
    # the day one does, rather than emitting every combination up front.
    if isinstance(key, tuple) and len(key) > 1 and key[1] in (0, None):
        alias = (key[0], 0 if key[1] is None else None, *key[2:])
        # `setdefault`, so an entry an operation wrote itself is never overwritten
        record.setdefault(alias, value)

remove(obj)

Drop the feature from the migration.

Emptying the dict is the signal every caller already understands: apply_ops stops applying rules to it and the engine never validates it into a target model.

Source code in xmas_core/processing/transform/ops/single.py
def remove(self, obj: dict[str, Any]) -> None:
    """Drop the feature from the migration.

    Emptying the dict is the signal every caller already understands: `apply_ops` stops
    applying rules to it and the engine never validates it into a target model.
    """
    if (feature_id := obj.get("id")) is not None:
        self.removed.add(feature_id)
        self.record(obj, "removed", True)
    obj.clear()

Staged(cls, obj) dataclass

One feature between its source model and its target model.

obj is the data every operation works on, and cls is the class that data is to be validated as - the source class until the rules have run, the target class afterwards. A dict cannot answer each: or an is_a: guard, which is the whole reason the class travels beside it rather than in it.

When

Bases: BaseModel

A guard on a single operation: one predicate over one or more attributes.

Roughly a seventh of the rules are one or two operations behind a condition, and this is what keeps them data. A clause names one subject - an attribute, a list of them, or a class - and carries at most one predicate over it. Naming no predicate is the common case and means the attribute has a value:

when: {attr: refText}                               # present
when: {attr: refText, not: true}                    # absent
when: {attr: art, in: [...], not: true}             # none of these values
when: {attr: position, geom: [XP_Liniengeometrie]}  # a value of this geometry type
when: {attrs: [a, b]}                               # both present
when: {attrs: [a, b], any: true}                    # at least one present
when: {attrs: [a, b], any: true, not: true}         # at least one absent

Presence is is not None, so an attribute stated as an explicit false counts as present - which is the whole reason it is not spelled as a value test.

not: negates the predicate per attribute and any: swaps the quantifier, which is what lets six earlier keywords (set, not_in, all_set, any_set, all_unset, any_unset) collapse into two modifiers. An operation may carry a list of clauses, which are ANDed, and that is the whole of it: no or, no nesting. A conjunction is still a table row; the moment a guard can be arbitrarily composed the rule file stops being something anyone can read.

any_ = Field(default=False, alias='any') class-attribute instance-attribute

Quantify attrs with any rather than all.

geom = None class-attribute instance-attribute

Geometry types by their UML name (XP_Flaechengeometrie), as the source version's properties declare them.

Not the model classes: the generated models spell a geometry type out as the union it admits (Polygon | MultiPolygon) and keep its name only as the property's typename.

geom_types = None class-attribute instance-attribute

(loader) The model.appschema.definitions classes geom admits. Resolved from the source appschema, and a load error if stated.

is_a = None class-attribute instance-attribute

Class predicate; only a collection operation can evaluate it, checked at load.

not_ = Field(default=False, alias='not') class-attribute instance-attribute

Negate the predicate, for each attribute separately.

predicate property

Which predicate this clause carries, or None for the presence test.

subject property

The attributes this clause tests. Empty for the class predicate.

matches(source, cls=None)

Evaluate the clause against a feature dict or a model - both are addressable.

cls is what an is_a clause tests, for a caller holding data rather than a model. Only a collection operation passes it: a rule sees one feature's dict and the loader refuses is_a there.

Source code in xmas_core/processing/transform/ops/single.py
def matches(self, source: Any, cls: type | None = None) -> bool:
    """Evaluate the clause against a feature dict or a model - both are addressable.

    `cls` is what an `is_a` clause tests, for a caller holding data rather than a model.
    Only a collection operation passes it: a rule sees one feature's dict and the loader
    refuses `is_a` there.
    """
    if self.is_a is not None:
        # by name through the MRO, so a clause works against either appschema
        # without anything having to resolve it to class objects first
        names = {self.is_a} if isinstance(self.is_a, str) else set(self.is_a)
        chain = cls if cls is not None else type(source)
        return bool(names & {b.__name__ for b in chain.__mro__})

    def get(attr: str) -> Any:
        if isinstance(source, dict):
            return source.get(attr)
        return getattr(source, attr, None)

    results = (self._test(get(attr)) != self.not_ for attr in self.subject)
    return any(results) if self.any_ else all(results)

apply_ops(ops, obj, ctx)

Run ops against obj in order, stopping if one removes the feature.

Source code in xmas_core/processing/transform/ops/single.py
def apply_ops(ops: list[Op], obj: dict[str, Any], ctx: RuleContext) -> None:
    """Run `ops` against `obj` in order, stopping if one removes the feature."""
    for op in ops:
        op.apply(obj, ctx)
        if not obj:
            return

run_collection_ops(ops, staged)

Run every operation in order over the staged features, which are mutated in place.

Nothing is validated here, in either slot. A pre operation writes into source-shaped data the rules are about to rewrite - one of them moves a Rasterdarstellung onto the plan's texte, where it becomes a TextAbschnitt in the same hop, which the source schema does not declare - and a post operation writes into target-shaped data the target models are about to be built from. Either way the check that matters is the one at the end of the hop, against the version the document is becoming.

Source code in xmas_core/processing/transform/ops/collection.py
def run_collection_ops(
    ops: list[CollectionOp], staged: dict[UUID, Staged]
) -> dict[UUID, Staged]:
    """Run every operation in order over the staged features, which are mutated in place.

    Nothing is validated here, in either slot. A `pre` operation writes into source-shaped
    data the rules are about to rewrite - one of them moves a Rasterdarstellung onto the
    plan's `texte`, where it becomes a TextAbschnitt in the same hop, which the source schema
    does not declare - and a `post` operation writes into target-shaped data the target
    models are about to be built from. Either way the check that matters is the one at the
    end of the hop, against the version the document is becoming.
    """
    for op in ops:
        op.run(staged)
    return staged

single

The declarative vocabulary a migration rule is written in.

A rule is a list of operations against the plain dict a source feature was dumped to. Every rule this package carries is one of eight - retype the feature, set, drop, move or map an attribute, fold several into a nested object, report, or remove the feature - so resources/migration/rules.yaml holds the whole migration as data. There is no escape hatch: an operation is handed the dict and nothing else, so it cannot reach another feature even by accident. The operations that legitimately need to, pull and push, live in collection and run in their own phase.

What an operation does not say is anything the target schema already does. Whether a moved value lands in a list or replaces one, whether a fold builds one object or several, and whether a field wants text are read off the target classes - by the loader where the rule file is the only thing that knows which class a feature becomes, and by the engine just before validation otherwise. That is what keeps eight operations enough.

Every operation that removes or moves an attribute also writes the migration record. That record is what the art-xpath remap follows, and having the operation write it is the point of the vocabulary: an earlier release had 55 hand-written bookkeeping calls next to the mutations they described, and they drifted.

CodeTable

Bases: BaseModel

How codes change: each listed code becomes its value, an unlisted one is kept or dropped.

One class for both places a table is written. A hop's rule for an enumeration is one, and the engine rewrites every property of that enumeration through it wherever it appears; a map operation is one rule's table for one attribute, optionally moving the result into another. A table value is one of:

  • a code, which takes the old one's place;
  • a list of codes, for an old code the new version covers with several, which take its place together;
  • null (or []), for a code with no successor, which goes the way an unlisted code does under unmapped: drop.

A table that replaces any code by several keeps every code once - BP_Wegerecht's 1000 and 2000 are 2000, 2500 and 2000 otherwise. A table that does not keeps a list's length, so a parallel detail list stays in step with it: two codes that became the same one are both kept, since collapsing them would hand the second code's detail to whatever came after it. That is also why the loader refuses paired beside the first kind.

A removed code is recorded as gone, and every entry that moves is recorded where it went.

remap(obj, ctx, attr, to=None, paired=None)

Rewrite attr through the table, in place or into to.

Into another attribute a mapped value moves, and an unmapped one stays where it was, so a drop after it reports what the table could not carry.

Source code in xmas_core/processing/transform/ops/single.py
def remap(
    self,
    obj: dict[str, Any],
    ctx: RuleContext,
    attr: str,
    to: str | None = None,
    paired: str | None = None,
) -> None:
    """Rewrite `attr` through the table, in place or into `to`.

    Into another attribute a mapped value *moves*, and an unmapped one stays where it
    was, so a `drop` after it reports what the table could not carry.
    """
    value = obj.get(attr)
    if value is None:
        return
    target = to or attr
    if isinstance(value, list):
        self._remap_list(obj, ctx, attr, target, paired)
        return
    if (mapped := self._lookup(value)) is _MISSING:
        return
    obj.pop(attr)
    if mapped is None:
        ctx.record(obj, (attr, None), None)
        return
    if target != attr:
        ctx.record(obj, (attr, None), (target, None))
    obj[target] = mapped

Drop

Bases: _BaseOp

Remove attributes the target version does not have.

Losing a value is the one thing a migration does that a reader needs told about, so a drop reports by default - once per attribute that actually held something, never for one that was already absent. That also makes it the way to report what an earlier operation left: a move with if_unset or a map that did not match leaves its source in place, and the drop after it reports exactly those.

warn: null silences it, for the drop of a value that is not lost: one a pre:/post: operation has already moved somewhere else.

Fold

Bases: _BaseOp

Collect several attributes into the nested object(s) the target version introduced.

A fold builds as many objects as its longest source has entries; a scalar source counts as one. Each source contributes its entry i to object i, a scalar source contributes to object [0] alone, and a source that runs out leaves the field unset. Where 5.x kept a code list and a parallel detail list side by side, the target pairs each code with its detail in one value, and pairing them by position is what those two lists always meant - a shared scalar beside them, a roof pitch over several roof shapes, is the same operation with one source that has nothing to pair past the first.

How many objects there are is therefore the data's answer, and whether to holds a list of them is the target's, filled in by the loader. A field inside an object is written as the value itself; the engine wraps it where the target declares the field 0..*. Reports to when it built something.

attrs = Field(min_length=1) class-attribute instance-attribute

Source attribute -> its field name inside the object. Two sources may name the same field, whose stream is then their values in the order they are written.

many = False class-attribute instance-attribute

(loader) to holds a list of objects rather than one.

Resolved from the target's own multiplicity, and a load error if a rule states it. It says where the objects go, never how many there are.

Map

Bases: CodeTable, _BaseOp

Rewrite one attribute's codes through a table, in place or into another attribute.

Distinct from a hop's rule for an enumeration only in reach: that rewrites every value of the enumeration wherever it appears, this one attribute of one rule.

paired = None class-attribute instance-attribute

A parallel detail list whose entry at a code's index goes with it.

A code replaced by nothing takes its detail with it. The loader refuses a code replaced by several, since it has no single position left to pair with.

Move

Bases: _BaseOp

Move values from one or more attributes to another.

What the move does with them is the target's answer, not the rule's: into a list they are appended after whatever to already holds, and into a single value the first one wins and the rest are lost. That one distinction is what rename, merge and take_first were three operations for. Reports the source whose entries were lost.

if_unset = False class-attribute instance-attribute

Leave an occupied to alone, and the source where it is.

The source is not consumed, so a drop after the move reports the value that did not move - which is what a version that folded two attributes into one and kept the more specific of them needs said.

many = False class-attribute instance-attribute

(loader) to holds a list. Resolved from the target, and a load error if stated.

set = None class-attribute instance-attribute

The field every moved value carries, with its value read from the from: table.

Written with item[field], because what a set: moves is a data type - XP_ExterneReferenz - and the engine dumps every data-typed property to a mapping before any rule runs. That is how attributes that each named one kind of document become one reference list tagged by typ.

Remove

Bases: _BaseOp

Drop the whole feature: the target version has no equivalent.

A requirement the target places on a value is a guard on this: a geometry of a type the target class has no place for is when: {attr: position, geom: [...], not: true}. A requirement that only reports is not an operation at all - an attribute the target declares 1..1 already fails validation, which says so.

abort = False class-attribute instance-attribute

Stop the whole migration: the document holds something with no mapping at all.

property = None class-attribute instance-attribute

The attribute that is the cause, checked at load like every other name a rule reads. Left out when the class itself is the cause, and the report then names the feature type.

warn = WarnCode.OBJECT_DROPPED class-attribute instance-attribute

null removes the feature without reporting, for one that is carried by another.

Retype

Bases: _BaseOp

Write the feature out as a different feature type, or one per plan type.

by_plan = None class-attribute instance-attribute

A target per plan type (BP, FP, ...), for a class the target split by plan.

targets property

Every class this retype can leave the feature as.

RuleContext(report, step, migration_record=dict(), removed=set(), feature=None, target_type=None, plan_prefix=None) dataclass

What an operation is handed besides the feature dict.

One instance per hop. It deliberately does not hold the collection: a rule operates on one feature, and the phase that may look at others gets a context of its own.

feature = None class-attribute instance-attribute

The feature currently being migrated, as (id, type); set by the engine per feature.

A data type has no id of its own, so a rule that warns about one would otherwise produce an issue no consumer could attach to anything. This is what it falls back to.

plan_prefix = None class-attribute instance-attribute

The document's plan type (BP, FP, ...), computed once per hop by the engine.

A property of the collection rather than of any rule, which is why it is resolved here for the by_plan of set and retype rather than stated in the rule file.

target_type = None class-attribute instance-attribute

The feature type the current feature will be validated as; retype writes it.

It is the feature's identity rather than one of its values, so it lives here instead of in the dict the operations mutate - where it sat beside the real attributes, could be read by a guard as though the feature declared such a property, and had to be popped back out before the dict could be validated. A data type has no type of its own here, so this is None while a rule reaches into one.

record(obj, key, value)

Note what happened to an attribute, so the art-xpath remap can follow it.

key and value are (name, index, ...) tuples in the shape parse_art_xpath produces; a value of None means the attribute is gone, and a list of tuples that it is several entries now.

The first entry of a repeatable attribute is filed under both spellings. xplan:typ and xplan:typ[1] name the same entry and a presentation object may be written either way, but parse_art_xpath keeps them apart as (typ, None) and (typ, 0) while enrich_attr_tuple puts them back together on the way out - so a record answering to only one of them silently left the other pointing at an attribute the target class no longer has. That was 58 art strings across 20 documents of tests/data, every one of them a fold source named without an index.

Source code in xmas_core/processing/transform/ops/single.py
def record(self, obj: dict[str, Any], key: Any, value: Any) -> None:
    """Note what happened to an attribute, so the `art`-xpath remap can follow it.

    `key` and `value` are `(name, index, ...)` tuples in the shape
    [`parse_art_xpath`][xmas_core.processing.style.parse_art_xpath] produces; a `value` of None
    means the attribute is gone, and a list of tuples that it is several entries now.

    The first entry of a repeatable attribute is filed under **both** spellings.
    `xplan:typ` and `xplan:typ[1]` name the same entry and a presentation object may be
    written either way, but `parse_art_xpath` keeps them apart as `(typ, None)` and
    `(typ, 0)` while `enrich_attr_tuple` puts them back together on the way out - so a
    record answering to only one of them silently left the other pointing at an
    attribute the target class no longer has. That was 58 `art` strings across 20
    documents of `tests/data`, every one of them a `fold` source named without an index.
    """
    record = self.migration_record.setdefault(obj.get("id"), {})
    record[key] = value
    # ponytail: the first index pair only. A deeper step could be spelled either way too,
    # but no operation writes a record key that varies past the first pair; widen this
    # the day one does, rather than emitting every combination up front.
    if isinstance(key, tuple) and len(key) > 1 and key[1] in (0, None):
        alias = (key[0], 0 if key[1] is None else None, *key[2:])
        # `setdefault`, so an entry an operation wrote itself is never overwritten
        record.setdefault(alias, value)

remove(obj)

Drop the feature from the migration.

Emptying the dict is the signal every caller already understands: apply_ops stops applying rules to it and the engine never validates it into a target model.

Source code in xmas_core/processing/transform/ops/single.py
def remove(self, obj: dict[str, Any]) -> None:
    """Drop the feature from the migration.

    Emptying the dict is the signal every caller already understands: `apply_ops` stops
    applying rules to it and the engine never validates it into a target model.
    """
    if (feature_id := obj.get("id")) is not None:
        self.removed.add(feature_id)
        self.record(obj, "removed", True)
    obj.clear()

Set

Bases: _BaseOp

Set an attribute: to a fixed value, another attribute's value, or a value per plan type.

Reports the attribute it set, for a value that is a guess a human should confirm.

by_plan = None class-attribute instance-attribute

A value per plan type (BP, FP, ...), for the few that differ by plan.

from_ = Field(default=None, alias='from') class-attribute instance-attribute

Copy this attribute's value, falling back to value when it is absent.

if_unset = False class-attribute instance-attribute

Leave an attribute that already holds a value alone.

Warn

Bases: _BaseOp

Report something about the feature without changing it.

When

Bases: BaseModel

A guard on a single operation: one predicate over one or more attributes.

Roughly a seventh of the rules are one or two operations behind a condition, and this is what keeps them data. A clause names one subject - an attribute, a list of them, or a class - and carries at most one predicate over it. Naming no predicate is the common case and means the attribute has a value:

when: {attr: refText}                               # present
when: {attr: refText, not: true}                    # absent
when: {attr: art, in: [...], not: true}             # none of these values
when: {attr: position, geom: [XP_Liniengeometrie]}  # a value of this geometry type
when: {attrs: [a, b]}                               # both present
when: {attrs: [a, b], any: true}                    # at least one present
when: {attrs: [a, b], any: true, not: true}         # at least one absent

Presence is is not None, so an attribute stated as an explicit false counts as present - which is the whole reason it is not spelled as a value test.

not: negates the predicate per attribute and any: swaps the quantifier, which is what lets six earlier keywords (set, not_in, all_set, any_set, all_unset, any_unset) collapse into two modifiers. An operation may carry a list of clauses, which are ANDed, and that is the whole of it: no or, no nesting. A conjunction is still a table row; the moment a guard can be arbitrarily composed the rule file stops being something anyone can read.

any_ = Field(default=False, alias='any') class-attribute instance-attribute

Quantify attrs with any rather than all.

geom = None class-attribute instance-attribute

Geometry types by their UML name (XP_Flaechengeometrie), as the source version's properties declare them.

Not the model classes: the generated models spell a geometry type out as the union it admits (Polygon | MultiPolygon) and keep its name only as the property's typename.

geom_types = None class-attribute instance-attribute

(loader) The model.appschema.definitions classes geom admits. Resolved from the source appschema, and a load error if stated.

is_a = None class-attribute instance-attribute

Class predicate; only a collection operation can evaluate it, checked at load.

not_ = Field(default=False, alias='not') class-attribute instance-attribute

Negate the predicate, for each attribute separately.

predicate property

Which predicate this clause carries, or None for the presence test.

subject property

The attributes this clause tests. Empty for the class predicate.

matches(source, cls=None)

Evaluate the clause against a feature dict or a model - both are addressable.

cls is what an is_a clause tests, for a caller holding data rather than a model. Only a collection operation passes it: a rule sees one feature's dict and the loader refuses is_a there.

Source code in xmas_core/processing/transform/ops/single.py
def matches(self, source: Any, cls: type | None = None) -> bool:
    """Evaluate the clause against a feature dict or a model - both are addressable.

    `cls` is what an `is_a` clause tests, for a caller holding data rather than a model.
    Only a collection operation passes it: a rule sees one feature's dict and the loader
    refuses `is_a` there.
    """
    if self.is_a is not None:
        # by name through the MRO, so a clause works against either appschema
        # without anything having to resolve it to class objects first
        names = {self.is_a} if isinstance(self.is_a, str) else set(self.is_a)
        chain = cls if cls is not None else type(source)
        return bool(names & {b.__name__ for b in chain.__mro__})

    def get(attr: str) -> Any:
        if isinstance(source, dict):
            return source.get(attr)
        return getattr(source, attr, None)

    results = (self._test(get(attr)) != self.not_ for attr in self.subject)
    return any(results) if self.any_ else all(results)

apply_ops(ops, obj, ctx)

Run ops against obj in order, stopping if one removes the feature.

Source code in xmas_core/processing/transform/ops/single.py
def apply_ops(ops: list[Op], obj: dict[str, Any], ctx: RuleContext) -> None:
    """Run `ops` against `obj` in order, stopping if one removes the feature."""
    for op in ops:
        op.apply(obj, ctx)
        if not obj:
            return

guards_last(data)

data with when and then moved to the end, in that order.

Source code in xmas_core/processing/transform/ops/single.py
def guards_last(data: dict[str, Any]) -> dict[str, Any]:
    """`data` with `when` and `then` moved to the end, in that order."""
    return {k: v for k, v in data.items() if k not in ("when", "then")} | {
        k: data[k] for k in ("when", "then") if k in data
    }

one_or_many_schema(schema)

Publish a when list as one clause or a list of them, as _one_or_many reads it.

Source code in xmas_core/processing/transform/ops/single.py
def one_or_many_schema(schema: dict[str, Any]) -> None:
    """Publish a `when` list as one clause or a list of them, as `_one_or_many` reads it."""
    items = schema.pop("items")
    del schema["type"]
    schema["anyOf"] = [items, {"type": "array", "items": items}]

value_types(names) cached

The value classes a geom: predicate was resolved to, looked up once.

Imported here rather than at module scope so the vocabulary stays independent of the model layer: every other part of it works on the dict alone.

Source code in xmas_core/processing/transform/ops/single.py
@functools.cache
def value_types(names: tuple[str, ...]) -> tuple[type, ...]:
    """The value classes a `geom:` predicate was resolved to, looked up once.

    Imported here rather than at module scope so the vocabulary stays independent of the
    model layer: every other part of it works on the dict alone.
    """
    from xmas_core.model.appschema import definitions

    return tuple(getattr(definitions, name) for name in names)

collection

The two operations that may look at more than one feature.

A migration rule works on one feature and is handed nothing but the dict it was dumped to, which is what makes single safe to read: no rule can reach into another feature, and none of them depends on the order features happen to appear in the document. A handful of migrations genuinely do need the neighbours, though - a Bereich that inherits verfahren from its plan, a TextAbschnitt that gains a role back to whatever references it - and this is where they live.

They are the same shape in both directions. pull follows association roles away from a feature and copies something back onto it; push follows the same roles and writes onto what it finds. Because the roles are named rather than expressed, the loader can check every one of them against the appschema, which is exactly what a path expression could not offer.

How either one writes is the written attribute's answer, not the rule's. Into a list they collect from every feature the roles reach, each value once; into a single value they fill it from the first feature that has one, and never overwrite what is there. The loader reads that off the appschema, so a rule states neither.

They run in two slots, declared as pre: and post: in the rule file, and both work on the same dumped feature data every other operation is handed: pre after the source models are dumped and before any rule runs, post after every rule has run and before the target models are built. Nothing runs in between, so neither can observe a half-migrated document.

What they move is data - a string, a UUID, a mapping - never a model. That is not a style preference: a live value carries its own class, and an enumeration member or a data type copied out of a source-version feature is a different class from the identically named one the target declares, so writing it onto a target model left the model holding a value its own field did not declare. Handing the data to the receiving model instead lets validation do the conversion, which it already knows how to do. It is also what lets a pre operation write an attribute the source class has no field for - verfahren is stated on the plan in 5.4 and on the Bereich in 6.0, and a dict has room for it where setattr on a 5.4 BP_Bereich raises.

Pull

Bases: _CollectionOp

Copy something from the features via reaches back onto the feature that owns it.

from: id copies the reached feature's own id, which is how a role pointing at it is filled in. An attrs: group is filled as one: only when the feature holds none of it, and all of it from the same reached feature.

attrs = None class-attribute instance-attribute

Shorthand for several attributes that keep their name on both sides.

with_ = Field(default_factory=dict, alias='with') class-attribute instance-attribute

Fixed fields added to each copied data type, rebuilt as the target role's type.

written property

The attributes this operation writes on the owning feature.

Push

Bases: _CollectionOp

Write onto the features via reaches, from the feature that owns the reference.

set = None class-attribute instance-attribute

Fixed values, for a role whose meaning moved onto the referenced feature.

to = None class-attribute instance-attribute

The role to write the owning feature's id to, which is how an inverse role is filled in.

Staged(cls, obj) dataclass

One feature between its source model and its target model.

obj is the data every operation works on, and cls is the class that data is to be validated as - the source class until the rules have run, the target class afterwards. A dict cannot answer each: or an is_a: guard, which is the whole reason the class travels beside it rather than in it.

run_collection_ops(ops, staged)

Run every operation in order over the staged features, which are mutated in place.

Nothing is validated here, in either slot. A pre operation writes into source-shaped data the rules are about to rewrite - one of them moves a Rasterdarstellung onto the plan's texte, where it becomes a TextAbschnitt in the same hop, which the source schema does not declare - and a post operation writes into target-shaped data the target models are about to be built from. Either way the check that matters is the one at the end of the hop, against the version the document is becoming.

Source code in xmas_core/processing/transform/ops/collection.py
def run_collection_ops(
    ops: list[CollectionOp], staged: dict[UUID, Staged]
) -> dict[UUID, Staged]:
    """Run every operation in order over the staged features, which are mutated in place.

    Nothing is validated here, in either slot. A `pre` operation writes into source-shaped
    data the rules are about to rewrite - one of them moves a Rasterdarstellung onto the
    plan's `texte`, where it becomes a TextAbschnitt in the same hop, which the source schema
    does not declare - and a `post` operation writes into target-shaped data the target
    models are about to be built from. Either way the check that matters is the one at the
    end of the hop, against the version the document is becoming.
    """
    for op in ops:
        op.run(staged)
    return staged

rules

Loading and validating the migration rules.

resources/migration/rules.yaml holds the whole migration as data: one entry of migrations: per hop, each with its rules - an operation list per feature or data type, a code table per enumeration - and its two collection phases, plus the templates: those entries share through YAML anchors. load reads one hop and checks it against the appschemas it claims to migrate between.

That check is the reason the rules are declared rather than discovered. An earlier release dispatched by getattr(self, "_" + ClassName.replace("_", "").lower()), so a rule whose name did not match any class was a silent no-op - which is what happened to _lpschutzobjektbundsrecht, misspelled against LP_SchutzobjektBundesrecht and never run in any release. Here that is a RuleFileError the moment the file loads.

RuleFileError

Bases: ValueError

A rule file does not describe the rules it claims to.

RuleSet(from_version, to_version, source, target, rules, enums, pre, post) dataclass

Everything one hop needs: its rules, its enum tables and its collection passes.

step property

The hop, as it is tagged onto every issue the hop raises.

build(text, from_version, to_version)

Validate rule-file text against the two appschemas and build the rule set.

Split out from load so the checks can be exercised against a rule file that does not have to exist on disk.

Parameters:

Name Type Description Default
text str

The rule file's contents.

required
from_version str

The version being migrated away from.

required
to_version str

The version being migrated to.

required

Raises:

Type Description
RuleFileError

The file names a class, enumeration, role or attribute that does not exist, or holds the hop asked for not exactly once.

Returns:

Type Description
RuleSet

The hop's rules, ready to run.

Source code in xmas_core/processing/transform/rules.py
def build(text: str, from_version: str, to_version: str) -> RuleSet:
    """Validate rule-file `text` against the two appschemas and build the rule set.

    Split out from [`load`][xmas_core.processing.transform.rules.load] so the checks can be exercised
    against a rule file that does not have to exist on disk.

    Args:
        text: The rule file's contents.
        from_version: The version being migrated away from.
        to_version: The version being migrated to.

    Raises:
        RuleFileError: The file names a class, enumeration, role or attribute that does
            not exist, or holds the hop asked for not exactly once.

    Returns:
        The hop's rules, ready to run.
    """
    hop = f"{from_version} -> {to_version}"
    document = _RuleDocument.model_validate(load_yaml(text) or {})
    # a list has no duplicate-key check to lean on, so a hop stated twice is looked for here
    matching = [
        m for m in document.migrations if (m.from_, m.to) == (from_version, to_version)
    ]
    if not matching:
        raise RuleFileError(f"rules.yaml holds no migration {hop}")
    if len(matching) > 1:
        raise RuleFileError(f"rules.yaml declares the migration {hop} twice")
    (parsed,) = matching

    source = Appschema.from_prefix("xplan", from_version)
    target = Appschema.from_prefix("xplan", to_version)

    errors: list[str] = []
    rules: dict[str, list[Op]] = {}
    enums: dict[str, CodeTable] = {}
    for name, rule in parsed.rules.items():
        cls = getattr(source.module, name, None)
        if isinstance(cls, type) and issubclass(cls, BaseEnum):
            if isinstance(rule, CodeTable):
                enums[name] = rule
            else:
                errors.append(
                    f"{name!r} is an enumeration, so its rule is a code table"
                )
            continue
        if not (isinstance(cls, type) and issubclass(cls, (FeatureType, DataType))):
            errors.append(
                f"{name!r} is not a feature type, data type or enumeration of xplan "
                f"{from_version}"
            )
            continue
        if not isinstance(rule, list):
            errors.append(
                f"{name!r} is a feature or data type, so its rule is a list of operations"
            )
            continue
        ops = rule
        errors += _check_rule(name, cls, ops, target, to_version)
        # a rule runs after the rules of every class it derives from, so what those wrote
        # is there to be read
        inherited = {
            attr
            for base in cls.__mro__[1:]
            for op, _ in _walk_ops(_ops_of(parsed.rules.get(base.__name__)))
            for attr in _WRITES.get(op.op, lambda _: [])(op)
        }
        live = _reachable_attrs(source, name) | inherited
        errors += _check_attrs(name, ops, live, from_version)
        ops, geom_errors = _resolve_geom(name, ops, source)
        errors += geom_errors
        rules[name], resolve_errors = _resolve_targets(name, ops, target, {name})
        errors += resolve_errors
    collection_ops: dict[str, list[CollectionOp]] = {"pre": [], "post": []}
    for slot, reading, writing in (
        # `pre` reads source-shaped data and `post` target-shaped data, but what either one
        # writes is only checked at the end of the hop, against the target. So a `pre`
        # operation may write an attribute the target introduces - `verfahren` moved from the
        # plan to the Bereich in 6.0 - as well as one the source states and the rules are
        # about to rewrite, and both halves have to be offered.
        ("pre", source, (source, target)),
        ("post", target, (target,)),
    ):
        for op in getattr(parsed, slot):
            resolved, op_errors = _check_collection_op(op, reading, writing)
            collection_ops[slot].append(resolved)
            errors += [f"{slot}: {e}" for e in op_errors]
    if errors:
        raise RuleFileError(
            f"migration {hop} does not match the appschemas it migrates between:\n"
            + "\n".join(f"- {e}" for e in errors)
        )

    return RuleSet(
        from_version=from_version,
        to_version=to_version,
        source=source,
        target=target,
        rules=rules,
        enums=enums,
        pre=collection_ops["pre"],
        post=collection_ops["post"],
    )

load(from_version, to_version) cached

Read, validate and return the rule set for one hop.

Parameters:

Name Type Description Default
from_version str

The version being migrated away from.

required
to_version str

The version being migrated to.

required

Raises:

Type Description
RuleFileError

The file names a class, enumeration, role or attribute that does not exist, or holds the hop asked for not exactly once.

Returns:

Type Description
RuleSet

The hop's rules, ready to run.

Source code in xmas_core/processing/transform/rules.py
@functools.cache
def load(from_version: str, to_version: str) -> RuleSet:
    """Read, validate and return the rule set for one hop.

    Args:
        from_version: The version being migrated away from.
        to_version: The version being migrated to.

    Raises:
        RuleFileError: The file names a class, enumeration, role or attribute that does
            not exist, or holds the hop asked for not exactly once.

    Returns:
        The hop's rules, ready to run.
    """
    return build(RULES_PATH.read_text("utf-8"), from_version, to_version)

engine

Running one migration hop.

Every source model is dumped once, up front, and everything after that works on the data:

  1. pre - the hop's collection operations over the source-shaped data, for the few migrations that need a feature's neighbours before it is rewritten.
  2. Per feature - apply the rules registered for every class in its inheritance chain, most abstract first, and point the feature at the class it is to be validated as.
  3. post - collection operations over the target-shaped data, once every rule has run.
  4. Repair - drop every reference to a feature the hop removed, or of a type its role no longer accepts.
  5. Validate - widen every scalar the target declares 0..* and render every value the target declares as text, turn each dict into a model of the target version and build the collection, then remap the presentation objects' art xpaths.

The separation is the point, and phase 2 is where it is enforced rather than described: a rule is handed the feature's dict and a RuleContext that does not hold the collection, so it cannot read another feature. An earlier release let rules reach into the half-built output while the loop was still running, so two of them carried a # order matters! comment and correctness depended on the order features happened to appear in the source document.

Phases 1 and 3 are the ones that may look at a neighbour, and both of them move data rather than models - see ops.collection. Nothing is validated until phase 5, which is what lets a pre operation write an attribute the source class has no field for and a post operation write one whose value only the target version knows how to read.

run_hop(collection, rules, report)

Migrate collection one version forward.

Parameters:

Name Type Description Default
collection FeatureCollection

The collection, at rules.from_version.

required
rules RuleSet

The hop's rules, from load.

required
report MigrationReport

The report to collect into; shared across every hop of a run.

required

Raises:

Type Description
MigrationError

A feature or the collection failed to validate against the target version. The report holds what went wrong with which feature.

Returns:

Type Description
FeatureCollection

The collection, at rules.to_version.

Source code in xmas_core/processing/transform/engine.py
def run_hop(
    collection: FeatureCollection,
    rules: RuleSet,
    report: MigrationReport,
) -> FeatureCollection:
    """Migrate `collection` one version forward.

    Args:
        collection: The collection, at `rules.from_version`.
        rules: The hop's rules, from [`load`][xmas_core.processing.transform.rules.load].
        report: The report to collect into; shared across every hop of a run.

    Raises:
        MigrationError: A feature or the collection failed to validate against the target
            version. The report holds what went wrong with which feature.

    Returns:
        The collection, at `rules.to_version`.
    """
    logger.info(f"Migration to XPlanung {rules.to_version} started")
    staged = {
        feature.id: Staged(cls=type(feature), obj=_dump(feature))
        for feature in collection.features.values()
    }
    ctx = RuleContext(
        report=report,
        step=rules.step,
        plan_prefix=_plan_prefix(collection),
    )

    run_collection_ops(rules.pre, staged)

    for feature_id, entry in staged.items():
        logger.debug(f"Current feature: {entry.cls.get_name()} with id {feature_id}")
        ctx.feature = (feature_id, entry.cls.get_name())
        _apply_rules(entry.obj, entry.cls, ctx, rules)
        # an empty dict is a feature a rule removed; it is not carried any further
        if entry.obj:
            _retarget(entry, feature_id, ctx, rules)

    run_collection_ops(rules.post, staged)

    _repair_refs(staged, ctx)

    models: dict[UUID, FeatureType] = {}
    for feature_id, entry in staged.items():
        ctx.feature = (feature_id, entry.cls.get_name())
        if entry.obj and (model := _validate(entry, feature_id, ctx)) is not None:
            models[feature_id] = model

    _raise_if_failed(report)
    migrated = _remap_art_xpaths(
        ctx, _build_collection(models, collection.srid, ctx, rules)
    )

    logger.info(f"Migration to XPlanung {rules.to_version} complete")
    return migrated

style

Methods for handling XPlanung presentational objects.

Parse and serialize art XPath values and derive styling properties for presentational objects by matching art references against the style rules in resources/style/rules.yaml (see rules) to populate stylesheetId and schriftinhalt.

PresentationObject

Bases: Protocol

The fields every version's presentational object classes declare beside FeatureType.

isinstance against it narrows a feature to one, which is how a checker learns what the generated classes know and FeatureType does not.

add_style_properties(collection, *, to_text=False, always_populate_schriftinhalt=False)

Adds styling properties to every presentational object in a collection.

The collection-level counterpart of add_style_properties_to_feature, which it applies to each presentational object with the object that object presents. A feature that names no object, names more than one, or carries no art is skipped and logged; only a reference naming a feature the collection does not hold is an error, since the collection's own reference check would already have refused it.

The features are replaced in place, so the collection is modified rather than copied.

Parameters:

Name Type Description Default
collection FeatureCollection

The collection whose presentational objects are styled.

required
to_text bool

Whether to convert symbolic presentational objects to textual ones.

False
always_populate_schriftinhalt bool

Populate schriftinhalt even if a rule has no text template.

False

Raises:

Type Description
ValueError

A presentational object references a feature that is not in the collection, or more than one style rule matched it.

Source code in xmas_core/processing/style/__init__.py
def add_style_properties(
    collection: FeatureCollection,
    *,
    to_text: bool = False,
    always_populate_schriftinhalt: bool = False,
) -> None:
    """Adds styling properties to every presentational object in a collection.

    The collection-level counterpart of
    [`add_style_properties_to_feature`][xmas_core.processing.style.add_style_properties_to_feature],
    which it applies to each presentational object with the object that object presents.
    A feature that names no object, names more than one, or carries no `art` is skipped
    and logged; only a reference naming a feature the collection does not hold is an error,
    since the collection's own reference check would already have refused it.

    The features are replaced in place, so the collection is modified rather than copied.

    Args:
        collection: The collection whose presentational objects are styled.
        to_text: Whether to convert symbolic presentational objects to textual ones.
        always_populate_schriftinhalt: Populate `schriftinhalt` even if a rule has no text
            template.

    Raises:
        ValueError: A presentational object references a feature that is not in the
            collection, or more than one style rule matched it.
    """
    logger.info("adding style properties to collection")

    for obj in collection.get_features():
        if not isinstance(obj, PresentationObject):
            continue
        if not obj.dientZurDarstellungVon:
            logger.info(f"Feature {obj.id}: dientZurDarstellungVon not set, skipping")
            continue
        elif len(obj.dientZurDarstellungVon) > 1:
            logger.warning(
                f"Feature {obj.id}: references to multiple objects '{obj.dientZurDarstellungVon}' not supported, skipping"
            )
            continue
        elif not obj.art:
            logger.info(f"Feature {obj.id}: art not set, skipping")
            continue
        ref_id = obj.dientZurDarstellungVon[0]
        if (
            not isinstance(ref_id, UUID)
            or (ref_obj := collection.features.get(ref_id)) is None
        ):
            raise ValueError(
                f"Feature {obj.id}: dientZurDarstellungVon references unknown feature {ref_id}"
            )
        new_obj = add_style_properties_to_feature(
            obj, ref_obj, to_text, always_populate_schriftinhalt
        )
        collection.features[obj.id] = new_obj

    logger.info("finished adding style properties to collection")

add_style_properties_to_feature(obj, ref_obj, to_text=False, always_populate_schriftinhalt=False)

Add styling properties to presentational objects.

This method parses object (dientZurDarstellungVon) and property (art) references from presentational objects and derives styling information (stylesheetId, schriftinhalt) based on a set of defined rules.

Parameters:

Name Type Description Default
obj FeatureType

The presentational object.

required
ref_obj FeatureType

The object referenced by the presentational object.

required
to_text bool

Whether to convert symbolic presentational objects to textual ones. Defaults to False.

False
always_populate_schriftinhalt bool

Populate schriftinhalt even if a rule has no text template.

False
Source code in xmas_core/processing/style/__init__.py
def add_style_properties_to_feature(
    obj: FeatureType,
    ref_obj: FeatureType,
    to_text: bool = False,
    always_populate_schriftinhalt: bool = False,
) -> FeatureType:
    """Add styling properties to presentational objects.

    This method parses object (dientZurDarstellungVon) and property (art) references from
    presentational objects and derives styling information (stylesheetId, schriftinhalt)
    based on a set of defined rules.

    Args:
        obj: The presentational object.
        ref_obj: The object referenced by the presentational object.
        to_text: Whether to convert symbolic presentational objects to textual ones. Defaults to False.
        always_populate_schriftinhalt: Populate `schriftinhalt` even if a rule has no text template.
    """
    uom_map = {"m2": "m²", "m3": "m³", "grad": "°"}

    def parse_art(ref_obj: FeatureType, art: str) -> dict[str, Any]:
        def parse_value(value: Any) -> dict[str, Any]:
            if prop_info.stereotype == "Measure":
                value = value.value
            if prop_info.typename == "Boolean":
                value = str(value).lower()

            if name.startswith("Z"):
                text = _to_roman(int(value))
            elif prop_info.stereotype == "Enumeration" and prop_info.enum:
                member = prop_info.enum(value)
                text = member.token or member.alias or member.label
            elif prop_info.stereotype == "Codelist":
                text = str(value)
                if text.startswith("urn:"):
                    text = text.split(f"urn:xplan:{prop_info.typename}:")[1]
            elif prop_info.stereotype == "Measure":
                uom = prop_info.uom or ""
                text = f"{value:n} {uom_map.get(uom, uom)}"
            else:
                text = value

            return {
                "value": value,
                "text": text,
            }

        xpath_input_tuple = parse_art_xpath(art)
        enriched_tuple, value, prop_info = enrich_attr_tuple(ref_obj, xpath_input_tuple)
        name = enriched_tuple[::2][-1]

        data = {
            "name": name,
            "data": parse_value(value),
            "type": prop_info.typename,
        }
        return data

    obj = deepcopy(obj)
    logger.debug(f"Feature {obj.id}: adding style properties")
    version = obj.appschema().version
    if to_text and (old_type := obj.get_name()) == "XP_PPO":
        new_type = "XP_PTO"
        obj = (
            obj.appschema()
            .model_factory(new_type, FeatureType)
            .model_validate(obj.model_dump())
        )
        logger.info(f"Feature {obj.id}: converted {old_type} to {new_type}")
    # TODO XPlanung v6.1: scale `skalierung` by the new massstabFaktor attribute

    if not isinstance(obj, PresentationObject):
        raise TypeError(
            f"Feature {obj.id}: {obj.get_name()} is not a presentational object"
        )
    logger.debug(
        f"parsing properties {obj.art} for referenced feature {ref_obj.get_name()} with ID {ref_obj.id}"
    )
    selectors = {}
    for art in obj.art or []:
        try:
            parsed_art = parse_art(ref_obj, art)
            selectors[parsed_art.pop("name")] = parsed_art
        except Exception:  # noqa: BLE001 - a broken art is logged and skipped
            logger.error(f"Feature {obj.id}: art '{art}' could not be parsed")
    valid_rules = []
    for rule_id, rule in rules.load().root.items():
        versioned_rule = rule.versions.get(version)
        if versioned_rule is None:
            continue
        # a rule without a selector matches nothing
        valid = (
            bool(versioned_rule.selector)
            and versioned_rule.selector.keys() == selectors.keys()
            and all(
                (
                    filter.value == ["*"]
                    or selectors[attr]["data"]["value"] in filter.value
                )
                and selectors[attr]["type"] == filter.type
                for attr, filter in versioned_rule.selector.items()
            )
        )
        if valid:
            valid_rules.append(str(rule_id))
            texts = {attr: data["data"]["text"] for attr, data in selectors.items()}
            obj.stylesheetId = AnyUrl(
                f"https://registry.gdi-de.org/codelist/de.xleitstelle.xplanung/XP_StylesheetListe/{rule_id}"
            )
            if (text := versioned_rule.text) and isinstance(obj, _TextObject):
                obj.schriftinhalt = text.format(**texts)
            elif always_populate_schriftinhalt and isinstance(obj, _TextObject):
                obj.schriftinhalt = " ".join(
                    [str(data["data"]["text"]) for data in selectors.values()]
                ).strip()
    if not valid_rules:
        logger.warning(f"No rule found for feature {obj.id}")
        obj.stylesheetId = None
        if isinstance(obj, _TextObject):
            obj.schriftinhalt = " ".join(
                [str(data["data"]["text"]) for data in selectors.values()]
            ).strip()
            logger.debug(f"Feature {obj.id}: schriftinhalt set to {obj.schriftinhalt}")
    if len(valid_rules) > 1:
        raise ValueError(f"More than one rules valid: {', '.join(valid_rules)}")
    else:
        logger.debug(f"Feature {obj.id}: stylesheetId set to {obj.stylesheetId}")
    return obj

enrich_attr_tuple(obj, art_tuple)

Return feature property information for construction of xpath expression.

The middle element of the result is whatever the last step of the walk landed on - a leaf attribute value (an enum member, a Measure, a string), not necessarily a model - so it is Any. Only the intermediate steps are models, and only a DataType can be one of those.

Source code in xmas_core/processing/style/__init__.py
def enrich_attr_tuple(
    obj: FeatureType, art_tuple: tuple[Any, ...]
) -> tuple[tuple[Any, ...], Any, PropertyInfo]:
    """Return feature property information for construction of xpath expression.

    The middle element of the result is whatever the last step of the walk landed on - a
    leaf attribute value (an enum member, a Measure, a string), not necessarily a model -
    so it is `Any`. Only the *intermediate* steps are models, and only a `DataType` can be
    one of those.
    """
    result = []

    current_obj = obj

    # iterate in pairs: (attr, index)
    for attr, idx in zip(art_tuple[::2], art_tuple[1::2]):
        property_info = current_obj.get_property_info(attr)

        if property_info.list:
            # a repeatable attribute is addressed by position, so a step that names one
            # without an index means its first entry and is written back saying so. The
            # cardinality decides this, not whoever produced the tuple - two migration
            # rules used to disagree about it and the same attribute came out spelled
            # both `xplan:detail` and `xplan:detail[1]`.
            idx = 0 if idx is None else idx
            current_obj = getattr(current_obj, attr)[idx]
        else:
            current_obj = getattr(current_obj, attr)
        result.extend([attr, idx])
        if isinstance(property_info.typename, list):
            current_type = current_obj.get_name()
        else:
            current_type = property_info.typename

        if property_info.stereotype == "DataType":
            result.extend([current_type, None])

    return tuple(result), current_obj, property_info

parse_art_xpath(xpath)

Parse an xpath expression into a tuple of feature property information and corresponding indices.

Source code in xmas_core/processing/style/__init__.py
def parse_art_xpath(xpath: str) -> tuple[Any, ...]:
    """Parse an xpath expression into a tuple of feature property information and corresponding indices."""
    result = []

    for segment in xpath.split("/"):
        match = _SEGMENT_PATTERN.match(segment)
        if not match:
            raise ValueError(f"Invalid segment: {segment}")

        name, index = match.groups()

        # Skip class markers like BP_KomplexeSondernutzung
        if _CLASS_PATTERN.match(name):
            continue

        result.append(name)
        result.append(max(int(index) - 1, 0) if index is not None else None)

    return tuple(result)

serialize_art_xpath(t, prefix='xplan')

Construct xpath expression from tuple of feature property information.

Source code in xmas_core/processing/style/__init__.py
def serialize_art_xpath(t: tuple[Any, ...], prefix: str = "xplan") -> str:
    """Construct xpath expression from tuple of feature property information."""
    if len(t) % 2 != 0:
        raise ValueError("Tuple must contain (name, index) pairs")

    parts = []

    for i in range(0, len(t), 2):
        name = t[i]
        index = t[i + 1]

        if not isinstance(name, str):
            raise TypeError(f"Expected str at position {i}, got {type(name)}")

        name = f"{prefix}:{name}"
        if index is None:
            parts.append(name)
        elif isinstance(index, int):
            parts.append(f"{name}[{index + 1}]")
        else:
            raise TypeError(
                f"Expected int or None at position {i + 1}, got {type(index)}"
            )

    return "/".join(parts)

rules

Loading and validating the style rules.

resources/style/rules.yaml holds every style rule as data, keyed by the stylesheetId it assigns. A rule states, per XPlanung version, which attributes a presentational object's art has to reference - their names, types and values - and optionally a template for its schriftinhalt. A version whose rule is the same as another version's is not written twice: the first is anchored and the others alias it, so they load as the same rule.

The models are the file's canonical form as well as its validation. Dumped with exclude_unset=True, they write fields in the order they declare them, only what was written, versions in ascending order and rules by comment in natural order, which is what scripts/normalize_style_rules.py writes back.

StyleRule

Bases: BaseModel

One stylesheetId: what it stands for and when it applies.

comment instance-attribute

Which Planzeichen the rule represents, or another note for whoever reads the file.

StyleRules

Bases: RootModel[dict[UUID, StyleRule]]

The rule file: every style rule, keyed by the stylesheetId it assigns.

StyleSelector

Bases: BaseModel

What one attribute referenced by art has to be.

type instance-attribute

Its type name in the version at hand.

value instance-attribute

The values it may hold, or ["*"] for any value.

VersionedStyleRule

Bases: BaseModel

A style rule as it applies to one XPlanung version.

selector instance-attribute

Keyed by attribute: art has to reference exactly these. A rule with no selector matches nothing.

text = None class-attribute instance-attribute

A template for schriftinhalt, with the selected attributes as placeholders.

load() cached

Read and validate the style rules, once.

Raises:

Type Description
RuleFileError

A key is written twice.

ValidationError

The file does not describe style rules.

Returns:

Type Description
StyleRules

The rules, with every alias resolved.

Source code in xmas_core/processing/style/rules.py
@functools.cache
def load() -> StyleRules:
    """Read and validate the style rules, once.

    Raises:
        RuleFileError: A key is written twice.
        pydantic.ValidationError: The file does not describe style rules.

    Returns:
        The rules, with every alias resolved.
    """
    return StyleRules.model_validate(load_yaml(RULES_PATH.read_text("utf-8")))