Merge pull request 'feat: say whether a host is still in its first boot' (#6) from feat/first-boot-pending into main

first_boot_pending() is true while /firstboot exists on FreeBSD. A FreeBSD
cloud image upgrades its base system on its first boot, starts sshd only
after that and restarts right away, and /etc/rc removes /firstboot just
before that restart. netOrk waits for this before it sets up a new VM
(NetOrk/netork#795).

OpenBSD has no such marker to ask yet, so it returns False there.

CHANGELOG.md

@@ -8,22 +8,6 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 ## [Unreleased]
 
 ### Added
-- SNMP: `run_device_action("fix_snmp")` installs net-snmp, writes the same
-  configuration netOrk uses on Linux, starts the agent and asks it for
-  sysDescr; `get_snmp_config` reports a running agent's community and port.
-
-### Changed
-- FreeBSD's `install_package` bootstraps pkg on a classic system that never
-  had it (`ASSUME_ALWAYS_YES`) instead of waiting for an answer.
-- Packages: `get_packages`, `install_package`, `uninstall_package`.
-- Updates: `get_available_updates`, with VuXML's verdict as `security` on
-  FreeBSD and base-system patches as one `base-system` entry (OpenBSD
-  `syspatch`, classic FreeBSD `freebsd-update`).
-- Services: `get_services` and `manage_service` (start, stop, restart,
-  enable, disable) through `service` and `rcctl`.
-- `get_listening_sockets()` in the shape of `ListeningSocketsMixin`: FreeBSD
-  through `sockstat`, OpenBSD through `fstat` as root and `netstat -an`
-  without, which the reading reports as `attributed: False`.
 - `FreeBSDDriver` (`freebsd`) and `OpenBSDDriver` (`openbsd`) on a shared
   `BsdDriver`: SSH over exec channels (no PTY, real exit codes), root through
   `sudo -S` with the password on stdin.
@@ -32,5 +16,24 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
   their addresses (`ifconfig -a`), routes (`netstat -rn`), ARP (`arp -an`),
   users and groups, processes (`ps … lstart`), cron jobs (system crontab and
   the login user's).
+- `get_listening_sockets()` in the shape of `ListeningSocketsMixin`: FreeBSD
+  through `sockstat`, OpenBSD through `fstat` as root and `netstat -an`
+  without, which the reading reports as `attributed: False`.
+- Packages: `get_packages`, `install_package`, `uninstall_package`. FreeBSD's
+  install bootstraps pkg on a classic system that never had it.
+- Updates: `get_available_updates`, with VuXML's verdict as `security` on
+  FreeBSD and base-system patches as one `base-system` entry (OpenBSD
+  `syspatch`, classic FreeBSD `freebsd-update`). A reader that cannot read
+  raises instead of returning no updates (#4).
+- Services: `get_services` and `manage_service` (start, stop, restart,
+  enable, disable) through `service` and `rcctl`.
+- SNMP: `run_device_action("fix_snmp")` installs net-snmp, writes the same
+  configuration netOrk uses on Linux, starts the agent and asks it for
+  sysDescr; `get_snmp_config` reports a running agent's community and port.
+- `get_host_status` (napalm-device-types' `HostStatusMixin`): on FreeBSD,
+  "reboot required" when the installed kernel differs from the running one.
+- `first_boot_pending()`: whether a FreeBSD host is still in its first boot
+  (`/firstboot`), so netOrk can wait before setting up a new VM
+  (NetOrk/netork#795).
 - Parsers tested on output recorded from FreeBSD 15.1 (hand-installed and
   cloud image) and OpenBSD 7.9 (NetOrk/netork#793).

README.md

@@ -37,6 +37,8 @@ over SSH, built on [napalm-device-types](https://git.netork.io/NAPALM/napalm-dev
 | `get_available_updates` | ✓ | ✓ | `pkg upgrade -n` + `pkg audit` (VuXML); `pkg_add -u -n -v` + `syspatch -c` |
 | `get_services`, `manage_service` | ✓ | ✓ | `service -e` + `service … status` / `service … <action>`; `rcctl ls on` + `rcctl check` / `rcctl <action>` |
 | `run_device_action("fix_snmp")`, `get_snmp_config` | ✓ | ✓ | net-snmp from packages: `/usr/local/etc/snmp/snmpd.conf` + `service snmpd`; `/etc/snmp/snmpd.conf` + `rcctl … netsnmpd` |
+| `get_host_status` | ✓ | unknown | napalm-device-types' host status: on FreeBSD `freebsd-version -k` vs `-r` (4.1+) |
+| `first_boot_pending` | ✓ | always `False` | whether `/firstboot` still exists |
 
 `get_listening_sockets` has the shape of napalm-device-types'
 `ListeningSocketsMixin` (whose `ss`/cgroup reading is Linux's) and its rule:
@@ -58,6 +60,20 @@ reports what `freebsd-update` has fetched (`updatesready`). On FreeBSD with
 pkgbase the base system is packages from the `FreeBSD-base` repository and
 needs no extra entry.
 
+A reader that cannot read **raises** rather than return an empty list: netOrk
+takes `[]` as "no updates" and would close every patch clock on the host. That
+covers a refused sudo, a failing `pkg`, `syspatch` or `freebsd-update`, and a pkg
+database pkg cannot read (only pkg's "not installed" means no packages).
+
+**Reboot.** `get_host_status` reports a FreeBSD host whose installed kernel
+(`freebsd-version -k`) differs from the running one (`-r`) as needing a reboot.
+OpenBSD has no such reading yet, so it stays unknown there.
+
+**First boot.** A FreeBSD cloud image upgrades its base system on its first boot,
+starts sshd only after that and restarts right away. `/etc/rc` removes `/firstboot`
+just before the restart, so `first_boot_pending()` is true until then; netOrk waits
+for it before it sets up a new VM. OpenBSD has no such marker to ask yet.
+
 **Services** are the enabled ones. FreeBSD reads their status as root, as
 root-only pidfiles hide a daemon from anyone else, and without root when sudo
 refuses; OpenBSD's `rcctl check` needs no root. Actions run as root.

napalm_bsd/base.py

@@ -17,6 +17,7 @@ from typing import Any, Optional
 import paramiko
 from napalm.base.exceptions import ConnectionClosedException, ConnectionException
 from napalm_device_types import OSDriver
+from napalm_device_types.host_status import HostStatusMixin
 from napalm_device_types.channel import (
     ByteStream,
     CommandResult,
@@ -47,7 +48,7 @@ _SNMPD_CONF = (
 _SNMP_TYPES = ("STRING:", "INTEGER:", "OID:", "Timeticks:", "Hex-STRING:", "IpAddress:")
 
 
-class BsdDriver(OSDriver):
+class BsdDriver(HostStatusMixin, OSDriver):
     """Base for the BSD drivers; a concrete one names the commands that differ."""
 
     VENDOR = ""
@@ -163,11 +164,53 @@ class BsdDriver(OSDriver):
         line, prefix = self._privileged(command, privileged)
         return open_stream_on_transport(self._transport(), line, stdin_prefix=prefix)
 
+    def _read(
+        self,
+        command: str,
+        *,
+        privileged: bool = False,
+        timeout: float = 60,
+        ok: tuple[int, ...] = (0,),
+    ) -> CommandResult:
+        """Run a reader's command; raise when it did not read.
+
+        A reader that cannot read raises rather than return "nothing": an empty
+        update list would close every patch clock netOrk keeps (napalm-bsd#4).
+        """
+        result = self.run_command(command, privileged=privileged, timeout=timeout)
+        if result.exit_code not in ok:
+            reason = (result.stderr or result.stdout).strip() or f"exit {result.exit_code}"
+            raise RuntimeError(f"{command}: {reason}")
+        return result
+
     def _out(self, command: str, *, privileged: bool = False, timeout: float = 60) -> str:
         """What *command* printed. A failing command prints nothing useful,
         and the readers below treat empty output as "nothing there"."""
         return self.run_command(command, privileged=privileged, timeout=timeout).stdout.strip()
 
+    #: A file that exists until the host's first boot is over; "" when there is none to ask.
+    FIRST_BOOT_MARKER = ""
+
+    def first_boot_pending(self) -> bool:
+        """Whether the host is still in its first boot.
+
+        A FreeBSD cloud image upgrades its base system on the first boot, starts
+        sshd only after that and restarts right away; ``/etc/rc`` removes
+        ``/firstboot`` just before the restart. netOrk waits for this before it
+        sets up a new VM (NetOrk/netork#795).
+        """
+        if not self.FIRST_BOOT_MARKER:
+            return False
+        return self.run_command(f"test -e {self.FIRST_BOOT_MARKER}").exit_code == 0
+
+    def _run_host_status_command(self, command: str) -> str:
+        """The transport for ``HostStatusMixin.get_host_status``: read-only, no root.
+
+        On FreeBSD the report compares ``freebsd-version -k`` with ``-r``
+        (napalm-device-types 4.1); elsewhere "reboot required" stays unknown.
+        """
+        return self.run_command(command).stdout
+
     # -- facts -------------------------------------------------------------------
 
     def _platform(self) -> dict[str, str]:

napalm_bsd/freebsd.py

@@ -31,6 +31,8 @@ class FreeBSDDriver(BsdDriver):
     ROUTES_COMMAND = "netstat -rnW"
     # sockstat names the process of every socket it can see; root sees them all.
     LISTENING_COMMAND = "sockstat -46lq -P tcp,udp"
+    # /etc/rc removes it at the end of the first boot.
+    FIRST_BOOT_MARKER = "/firstboot"
 
     def _parse_listening(self, output: str, *, attributed: bool) -> list[dict]:
         return parse.sockstat(output)
@@ -53,8 +55,18 @@ class FreeBSDDriver(BsdDriver):
         return parse.service_status(output)
 
     def _has_pkg(self) -> bool:
-        """pkg is bootstrapped; a hand-installed classic system may have only the stub."""
-        return self.run_command("pkg -N").exit_code == 0
+        """pkg is bootstrapped; a hand-installed classic system may have only the stub.
+
+        Only the stub's "not installed" means no pkg; any other failure (a
+        database pkg cannot read) raises rather than read as "no packages".
+        """
+        result = self.run_command("pkg -N")
+        if result.exit_code == 0:
+            return True
+        message = (result.stderr or result.stdout).strip()
+        if "is not installed" in message:
+            return False
+        raise RuntimeError(f"pkg -N: {message or f'exit {result.exit_code}'}")
 
     def get_packages(self) -> list[dict]:
         return parse.pkg_query(self._out(self.PKG_QUERY)) if self._has_pkg() else []
@@ -69,10 +81,13 @@ class FreeBSDDriver(BsdDriver):
         """
         updates: list[dict] = []
         if self._has_pkg():
-            updates = parse.pkg_upgrades(self._out("pkg upgrade -n", privileged=True, timeout=300))
+            upgrade = self._read("pkg upgrade -n", privileged=True, timeout=300)
+            updates = parse.pkg_upgrades(upgrade.stdout)
             audit = self.run_command("pkg audit -Fq", privileged=True, timeout=120)
-            known = audit.exit_code in (0, 1)  # 1: vulnerable packages found
-            vulnerable = parse.pkg_audit(audit.stdout) if known else set()
+            vulnerable = parse.pkg_audit(audit.stdout)
+            # 1 is "vulnerable packages found" and any error alike; only a list
+            # of packages makes it a verdict.
+            known = audit.exit_code == 0 or (audit.exit_code == 1 and bool(vulnerable))
             for update in updates:
                 update["security"] = (update["name"] in vulnerable) if known else None
             if "FreeBSD-base" in self._out("pkg repos -l").split():
@@ -86,10 +101,13 @@ class FreeBSDDriver(BsdDriver):
         2 when there are none; it does not fetch, which ``freebsd-update cron``
         does daily where it is enabled.
         """
-        ready = self.run_command(
-            "freebsd-update --not-running-from-cron updatesready", privileged=True, timeout=60
+        ready = self._read(
+            "freebsd-update --not-running-from-cron updatesready",
+            privileged=True,
+            timeout=60,
+            ok=(0, 2),
         )
-        if ready.exit_code != 0:
+        if ready.exit_code == 2:
             return []
         return [
             {

napalm_bsd/openbsd.py

@@ -64,10 +64,10 @@ class OpenBSDDriver(BsdDriver):
         updates: list[dict] = [
             {**candidate, "origin": None, "security": None}
             for candidate in parse.pkg_add_candidates(
-                self._out("pkg_add -u -n -v", privileged=True, timeout=300)
+                self._read("pkg_add -u -n -v", privileged=True, timeout=300).stdout
             )
         ]
-        patches = parse.syspatch(self._out("syspatch -c", privileged=True, timeout=120))
+        patches = parse.syspatch(self._read("syspatch -c", privileged=True, timeout=120).stdout)
         if patches:
             installed = parse.syspatch(self._out("syspatch -l", privileged=True))
             summary = (

tests/test_drivers.py

@@ -267,6 +267,15 @@ class TestPackages:
         assert driver.get_packages() == []
         assert driver.calls == [("pkg -N", False)]
 
+    def test_a_pkg_that_cannot_read_its_database_raises(self):
+        """Any other failure is "could not read", never "no packages"."""
+        driver = _channel(
+            FreeBSDDriver,
+            {"pkg -N": ("pkg: sqlite error ...: database disk image is malformed", 1)},
+        )
+        with pytest.raises(RuntimeError, match="malformed"):
+            driver.get_packages()
+
     def test_openbsd_packages(self):
         driver = _channel(OpenBSDDriver, {"pkg_info": _read("openbsd-vm", "pkg_info_full.txt")})
         assert any(p["name"] == "pcre2" for p in driver.get_packages())
@@ -359,7 +368,7 @@ class TestAvailableUpdates:
         driver = _channel(
             FreeBSDDriver,
             {
-                "pkg -N": ("", 1),
+                "pkg -N": ("pkg: pkg is not installed", 1),
                 ("freebsd-update --not-running-from-cron updatesready", True): ("", 2),
             },
         )
@@ -532,3 +541,95 @@ class TestSnmp:
     def test_no_agent_running_means_no_config(self):
         driver = _channel(OpenBSDDriver, {"pgrep -f /usr/local/sbin/snmpd": ("", 1)})
         assert driver.get_snmp_config() is None
+
+
+class TestHostStatus:
+    """napalm-device-types' host status, carried over the exec channel; on
+    FreeBSD it compares the installed with the running kernel (4.1.0)."""
+
+    REPORT = (
+        "HSTAT_BEGIN\n[kernel]\n15.1-RELEASE\n[modules]\n"
+        "[freebsd-kernel]\n15.1-RELEASE-p5\n[freebsd-running]\n15.1-RELEASE-p4\n"
+        "[timers]\nHSTAT_END\n"
+    )
+
+    def test_a_patched_kernel_waits_for_a_reboot(self):
+        from napalm_device_types.host_status import HOST_STATUS_COMMAND
+
+        driver = _channel(FreeBSDDriver, {HOST_STATUS_COMMAND: self.REPORT})
+        status = driver.get_host_status()
+        assert status["reboot_required"] is True
+        assert driver.calls == [(HOST_STATUS_COMMAND, False)]  # read-only, no root
+
+    @pytest.mark.parametrize("cls", [FreeBSDDriver, OpenBSDDriver])
+    def test_both_have_it(self, cls):
+        assert callable(getattr(cls, "get_host_status", None))
+
+
+class TestAFailedReadRaises:
+    """A reader that could not read raises; [] would tell netOrk "no updates"
+    and close every patch clock on the host (napalm-bsd#4)."""
+
+    def test_freebsd_without_root(self):
+        driver = _channel(
+            FreeBSDDriver,
+            {"pkg -N": "", ("pkg upgrade -n", True): ("", 1)},
+        )
+        with pytest.raises(RuntimeError, match="pkg upgrade -n"):
+            driver.get_available_updates()
+
+    def test_a_failed_audit_leaves_security_unknown(self):
+        """pkg audit exits 1 for "vulnerable packages found" and for errors alike;
+        only a list of packages makes the 1 a verdict."""
+        driver = _channel(
+            FreeBSDDriver,
+            {
+                "pkg -N": "",
+                ("pkg upgrade -n", True): _read("freebsd-vm", "sudo_pkg_upgrade_n_latest.txt"),
+                ("pkg audit -Fq", True): ("", 1),
+                "pkg repos -l": "FreeBSD-ports\nFreeBSD-base\n",
+            },
+        )
+        assert {u["security"] for u in driver.get_available_updates()} == {None}
+
+    def test_classic_freebsd_update_error(self):
+        driver = _channel(
+            FreeBSDDriver,
+            {
+                "pkg -N": ("pkg: pkg is not installed", 1),
+                ("freebsd-update --not-running-from-cron updatesready", True): ("", 1),
+            },
+        )
+        with pytest.raises(RuntimeError, match="freebsd-update"):
+            driver.get_available_updates()
+
+    @pytest.mark.parametrize("failing", ["pkg_add -u -n -v", "syspatch -c"])
+    def test_openbsd_without_root(self, failing):
+        answers = {("pkg_add -u -n -v", True): "", ("syspatch -c", True): ""}
+        answers[(failing, True)] = ("", 1)
+        driver = _channel(OpenBSDDriver, answers)
+        with pytest.raises(RuntimeError, match=failing.split()[0]):
+            driver.get_available_updates()
+
+
+class TestFirstBoot:
+    """Whether the host is still in its first boot (NetOrk/netork#795).
+
+    A FreeBSD cloud image upgrades its base system on the first boot, starts
+    sshd only then and restarts right after; /etc/rc removes /firstboot just
+    before that restart. netOrk waits for it before setting up a new VM.
+    """
+
+    def test_freebsd_in_its_first_boot(self):
+        driver = _channel(FreeBSDDriver, {"test -e /firstboot": ("", 0)})
+        assert driver.first_boot_pending() is True
+        assert driver.calls == [("test -e /firstboot", False)]
+
+    def test_freebsd_after_its_first_boot(self):
+        driver = _channel(FreeBSDDriver, {"test -e /firstboot": ("", 1)})
+        assert driver.first_boot_pending() is False
+
+    def test_openbsd_has_no_marker_to_ask(self):
+        driver = _channel(OpenBSDDriver, {})
+        assert driver.first_boot_pending() is False
+        assert driver.calls == []