Compare commits

..
10 Commits
Author SHA1 Message Date
KeelrLeek cb8dbc5f2e README: front page explaining what Harbor is, why it helps agent teams, how it works, and a quick start 2026-10-09 12:32:25 -05:00
KeelrLeek 56308e3aab Background compiles give way to the lander: the Logbook statistics and the results refresher wait while [logbook] lander_lock is fresh (under 60 min); the statistics tools run below normal priority and a timeout ends the whole tree (an orphaned cl.exe went on compiling main at normal priority); a retried statistics run no longer relogs the same commits 2026-10-09 10:46:40 -05:00
KeelrLeek 21b7fedb6f re_summary: one call for a function's name, prototype, size, convention, readable status, callers and callees with their status, globals, strings, vtable slots and Mac/Xbox hints (harbor re summary); ghidra_check: confirm the shared Ghidra server serves our image (address range, byte samples, SHA-256 when the imported file is readable) (harbor re ghidra-check) 2026-10-09 09:51:09 -05:00
KeelrLeek 57fb31f2d6 watch.py reports times in CST 2026-10-07 18:21:43 -05:00
KeelrLeek b28f80f1a4 Queue skips catch and unwind funclets (they open on the parent's frame); ruling #12777 2026-10-07 18:10:13 -05:00
KeelrLeek b8dab67871 Logbook clarity: game-code headline, CST times, one size unit, tier table from the progress map with a game column, Decomp view counts game functions, ship's log without repeated or cut titles, newest milestone in the banner; whole-program check builds a clean snapshot of main 2026-10-07 17:05:18 -05:00
KeelrLeek 783c46da56 Logbook: game-functions tile counts game functions only (was all-code decompiled over game total) 2026-10-07 16:48:43 -05:00
KeelrLeek 71e6d6cfda logbook: an All code bar (game + engine + platform, by bytes, with function counts) next to the game-code bar, so engine work shows too 2026-10-07 16:43:38 -05:00
KeelrLeek 59dac57bc8 claims: a draft claim inside another agent's band is allowed when a coordinator assigned the agent a task for that scope (reassigned rework) 2026-10-07 13:27:57 -05:00
KeelrLeek c77ed1a4db policy submits_per_hour: an agent can be capped at N submissions or resubmissions per hour (the User asked the DeepSeeks to slow down and check their work) 2026-10-07 11:04:54 -05:00
17 changed files with 708 additions and 52 deletions

No files matched your search

+83 -4
View File
@@ -1,6 +1,85 @@
# Harbor
<div align="center">
# ⚓ Harbor
**A local coordination hub for teams of AI coding agents, and the person running them.**
Agents message each other, split up the work, avoid stepping on each other, and review each other's changes. You watch it all from one page, and step in only when you want to.
`Python 3.11+` · `standard library only` · `SQLite` · `MCP` · `Claude Code` · `Codex` · `OpenCode` · `Kimi` · `DeepSeek`
</div>
---
## Why Harbor?
When several agents work on one codebase, they usually can only talk to each other through *you*. You copy questions from one window into another. You notice too late that two of them rewrote the same file. You can't see who is stuck, who is idle, or who quietly stopped an hour ago.
Harbor gives the team a shared place to talk and coordinate:
| Without Harbor | With Harbor |
|---|---|
| You relay every question between agents | Agents send threaded messages straight to each other, and acknowledge them |
| Two agents edit the same file | **Claims** lock a file, function or address range. A collision alert fires before anyone commits |
| "What should I do next?" | A **task queue** with priorities, dependencies and owners. An idle agent runs `next` |
| Agents approve their own work | A **review queue** where nobody reviews their own work, and only a coordinator lands it |
| A session ends and its context is lost | **Handoffs** and generated **restart prompts** let a fresh session carry on |
| You poll each window to see what's happening | A **live board**: presence, a tool-call feed, alerts, a team timeline, git status for every worktree |
| Stopping an agent means hunting for its window | **One stop switch** for everyone or for one agent, enforced on every tool call. Only the human resumes |
It's built for long-running, multi-agent work: hours or days, several apps at once, with the human dipping in and out.
## How it works
```mermaid
flowchart LR
subgraph Agents
A1[Claude Code]
A2[Codex]
A3[OpenCode / Kimi / DeepSeek]
end
A1 -- MCP tools --> H
A2 -- MCP tools --> H
A3 -- CLI / MCP --> H
A1 -. hooks: activity, stop switch .-> H
A2 -. hooks .-> H
H[(Harbor<br/>SQLite per project)]
H -- chat page + board --> U[You, in the browser]
U -- answers, stop / resume, tasks --> H
```
- **One SQLite database per project**, in WAL mode. The CLI, the web server and any number of MCP servers can all write to it safely at once.
- **Agents connect through MCP or the CLI.** `harbor install <host>` wires up Claude Code, Codex, OpenCode, Kimi or DeepSeek Harness, including hooks that feed live activity and enforce the stop switch.
- **Each agent's identity is fixed by its host configuration**, so an agent can't speak as another.
- **The web server binds to localhost only.** It gives you a texting-style chat page and a full coordination board.
## Quick start
```bash
git clone https://git.whoagland.com/Keelrleek/Harbor.git && cd Harbor
# create a project with two agents, one of them the coordinator
py -3.12 -B run.py init garden --title "Garden app" --agent Lead:"Builds it" --agent Helper --coordinator Lead --marker D:/code/garden
# start the chat page (http://127.0.0.1:8766) and the board (/board)
py -3.12 -B run.py serve --port 8766
# connect an agent app (codex | claude | opencode | kimi | deepseek)
py -3.12 -B run.py install claude --agent Lead
# check everything is wired up
py -3.12 -B run.py doctor
```
On Linux or macOS, use `python3` in place of `py -3.12`. To get a `harbor` command, run `pip install -e .`. The optional MCP server needs `pip install -e .[mcp]`.
Set your Harbor home (where project files and databases live) with `$HARBOR_HOME`, or with `home = '...'` in `local.toml`. The default is `~/.harbor`.
---
## Features
Harbor is a local coordination hub for a team of AI coding agents and the person running them. It works with Codex, Claude Code, OpenCode, Kimi, DeepSeek Harness, or anything that can run a command or speak MCP.
**Talking**
- **Threads.** Messages are grouped by subject (`fn:0x529350`, `commit:b4b44f0`, `task:5`, `item:12`), each with explicit recipients.
@@ -90,7 +169,7 @@ Harbor is a local coordination hub for a team of AI coding agents and the person
Each project is one SQLite database (WAL mode), so the CLI, the web server and many MCP servers can write at once safely. A project can also mirror every message into a plain `[time] From → To: text` file for older tools.
## Quick start
## Project setup
```bash
py -3.12 -B run.py init garden --title "Garden app" --agent Lead:"Builds it" --agent Helper --coordinator Lead --marker D:/code/garden
@@ -183,4 +262,4 @@ Hooks resolve the project from the agent's working folder and stay silent anywhe
py -3.12 -B -m unittest discover -s tests
```
The tests use temporary homes under `R:\Workspace\Data\openseas-chat\test-tmp`, or `$HARBOR_TEST_TMP`.
Set `$HARBOR_TEST_TMP` to a scratch folder first; the tests create their temporary homes there (the built-in default is the original author's path).
+16 -1
View File
@@ -255,7 +255,7 @@ def build_parser():
p = sub.add_parser("who", help="who has worked on a scope (notes, reviews, claims, messages)")
p.add_argument("scope")
p = sub.add_parser("re", help="reverse-engineering lookups from the project's image and annotations")
p.add_argument("action", choices=["disasm", "xrefs", "bytes", "lookup", "ghidra", "ghidra-tools", "debugger-tools", "match", "similar", "struct-audit", "behaviour"])
p.add_argument("action", choices=["disasm", "xrefs", "bytes", "lookup", "summary", "ghidra", "ghidra-tools", "ghidra-check", "debugger-tools", "match", "similar", "struct-audit", "behaviour"])
p.add_argument("target", nargs="?", default="", help="an address (0x4AD850) or a name; for ghidra, the tool")
p.add_argument("arguments", nargs="?", default="{}", help="ghidra: the tool's arguments as JSON")
p.add_argument("--decomp", help="match: the checkout or worktree to compile from")
@@ -722,6 +722,21 @@ def main(argv=None):
elif args.action == "ghidra-tools":
res = Ghidra(harbor.project.reverse).tools()
out(res, "\n".join(f"{t['name']:<26} {t['description'].splitlines()[0][:90]}" for t in res))
elif args.action == "ghidra-check":
from .reverse import ghidra_check
res = ghidra_check(harbor.project.reverse)
mark = lambda ok: "ok " if ok else ("?? " if ok is None else "FAIL")
out(res, "\n".join(
[f"{'Ghidra serves our program' if res['ok'] else 'Ghidra does NOT serve our program'}: "
f"{res['program']} ({res['functions_in_ghidra']} functions); image {res['image_base']}, "
f"{res['image_size']} bytes, sha256 {res['image_sha256'][:16]}..."] +
[f" {mark(c['ok'])} {c['check']}: " + (f"expected {c['expected']}, Ghidra {c['ghidra']}" if "ghidra" in c
else ("differs at " + ", ".join(c["differ_at"]) if c.get("differ_at") else "all equal"))
for c in res["checks"]]))
elif args.action == "summary":
from .reverse import format_summary
res = r.summary(args.target)
out(res, format_summary(res))
elif args.action == "ghidra":
text = Ghidra(harbor.project.reverse).call(args.target, json.loads(args.arguments))
out({"result": text}, text)
+11 -1
View File
@@ -709,7 +709,14 @@ class Harbor(WorkMixin, ExtrasMixin, FlowMixin, LaunchMixin):
if kind == "draft" and lo is not None:
bands = [a for a in active if a["kind"] == "band"
and scopes_overlap((canonical, lo, hi), (a["scope"], a["lo"], a["hi"]))]
if bands and not any(b["agent"] == agent for b in bands):
# a task a coordinator assigned to this agent for this scope is the exception to another's band
assigned = any(scopes_overlap((canonical, lo, hi), (t["scope"], *parse_scope(t["scope"])[1:]))
for t in c.execute("SELECT scope FROM tasks WHERE assignee=? AND scope IS NOT NULL "
"AND state IN ('open','in_progress') AND created_by IN (%s)"
% ",".join("?" * len(self.coordinators)),
(agent, *self.coordinators)).fetchall()
if t["scope"].startswith(("fn:", "range:", "file:")))
if bands and not any(b["agent"] == agent for b in bands) and not assigned:
conflicts += bands
if conflicts and not override:
described = "; ".join(f"{a['agent']} holds {a['kind']} {a['scope']}"
@@ -762,6 +769,7 @@ class Harbor(WorkMixin, ExtrasMixin, FlowMixin, LaunchMixin):
raise HarborError("no reviewer: name one, or set reviewers in the project file")
canonical = parse_scope(scope)[0] if scope else None
commit = self.resolve_commit(author, commit)
self.check_submit_policy(author)
def do(c):
now = self.now()
@@ -825,6 +833,8 @@ class Harbor(WorkMixin, ExtrasMixin, FlowMixin, LaunchMixin):
self.check_landing_policy(item_id)
if is_author and commit:
commit = self.resolve_commit(agent, commit)
if is_author and to_state == "submitted":
self.check_submit_policy(agent)
def do(c):
now = self.now()
+14
View File
@@ -265,6 +265,20 @@ class FlowMixin:
raise HarborError(f"policy: item #{item_id} needs approval from {', '.join(missing)} before it lands "
"(since its last submission)")
def check_submit_policy(self, author):
"""policy submits_per_hour: an agent capped at N submissions (new or resubmitted) per hour must slow down
and check its work before sending more."""
cap = (self.policy("submits_per_hour") or {}).get(author)
if not cap:
return
since = iso(utc(self.clock() - 3600))
count = self.conn.execute("SELECT COUNT(*) FROM item_events WHERE agent=? AND to_state='submitted' AND at>=?",
(author, since)).fetchone()[0]
if count >= int(cap):
raise HarborError(f"policy: at most {cap} submissions per hour for {author}. Slow down: re-read the "
"disassembly against your C line by line, run quick_readable and re_behaviour, list "
"your changed annotation rows, and submit only work you have checked twice")
def check_message_policy(self, sender, to_all):
cap = self.policy("all_per_hour")
if not cap or not to_all or sender in self.coordinators:
+69
View File
@@ -0,0 +1,69 @@
"""The server's background compiles (the Logbook's statistics, the whole-program check, the emulator run) give way to
the lander: its gates (a full readable check of every file) are what land work on main, and a second compile of main
beside them roughly quadruples their time.
Two parts:
lander_busy(project) while [logbook] lander_lock names a file that exists and was written in the last
STALE_MINUTES, the lander is running its gates: background jobs do not start. The lander
creates the file before its gates and removes it after the push; an older file is a lander
that died, and is ignored.
run_low(...) subprocess.run at BELOW_NORMAL priority (children such as cl.exe inherit it) that, on a
timeout, ends the whole process tree instead of leaving the compiler running orphaned.
Configured per project in the [logbook] table:
lander_lock = 'R:/Workspace/Data/lander-busy.lock' unset or "": never wait"""
import os
import subprocess
import time
from pathlib import Path
STALE_MINUTES = 60
BELOW_NORMAL = getattr(subprocess, "BELOW_NORMAL_PRIORITY_CLASS", 0)
IDLE = getattr(subprocess, "IDLE_PRIORITY_CLASS", 0)
NO_WINDOW = getattr(subprocess, "CREATE_NO_WINDOW", 0)
def lock_path(project):
path = ((getattr(project, "logbook", None) or {}).get("lander_lock") or "").strip()
return Path(path) if path else None
def lander_busy(project, now=None):
"""Why background work should wait (the lander's lock is fresh), or None."""
path = lock_path(project)
if path is None:
return None
try:
written = path.stat().st_mtime
except OSError:
return None
age = (time.time() if now is None else now) - written
if age >= STALE_MINUTES * 60:
return None
return f"the lander is running its gates ({path.name}, {max(0, int(age // 60))} min old)"
def kill_tree(pid):
if os.name == "nt":
subprocess.run(["taskkill", "/PID", str(pid), "/T", "/F"], capture_output=True, creationflags=NO_WINDOW)
else:
try:
os.kill(pid, 9)
except OSError:
pass
def run_low(argv, timeout, priority=BELOW_NORMAL, **kwargs):
"""subprocess.run(argv, capture_output=True, text=True, ...) at a lower priority class; on a timeout the whole
tree (cmd.exe, cl.exe) is ended before TimeoutExpired is raised."""
flags = kwargs.pop("creationflags", 0) | (priority | NO_WINDOW if os.name == "nt" else 0)
with subprocess.Popen(argv, stdin=subprocess.DEVNULL, stdout=subprocess.PIPE, stderr=subprocess.PIPE,
text=True, creationflags=flags, **kwargs) as proc:
try:
stdout, stderr = proc.communicate(timeout=timeout)
except subprocess.TimeoutExpired:
kill_tree(proc.pid)
proc.kill()
proc.communicate()
raise subprocess.TimeoutExpired(argv, timeout) from None
return subprocess.CompletedProcess(argv, proc.returncode, stdout, stderr)
+20 -7
View File
@@ -9,6 +9,8 @@ Configured per project in a [logbook] table:
decomp = 'R:/.../open-seas-decomp' the repository whose main branch is followed
stats_repo = 'R:/.../stats-work/repo' a scratch copy of open-seas holding tools/progress/*.py
interval_minutes = 10 how often the server looks (default 10)
lander_lock = 'R:/.../lander-busy.lock' wait while the lander runs its gates (harbor.lander_gate)
The progress tools run below normal priority.
Milestones and narrative entries stay with people: they need judgement.
State (the last commit seen) is kept in the logbook folder as auto.json."""
@@ -22,9 +24,10 @@ import time
import traceback
from pathlib import Path
from . import logbook
from . import lander_gate, logbook
MAX_ENTRIES = 25
STATS_TIMEOUT = 1800 # each progress tool; sync_decomp compiles every readable file, below normal priority
def _git(repo, *args):
@@ -73,8 +76,8 @@ def log_commits(project, repo, since, head):
for sha, when, subject, body in commits:
files = [f for f in _git(repo, "show", "--name-only", "--format=", sha).splitlines() if f.strip()]
lines = [l for l in body.strip().splitlines() if l.strip() and not l.lower().startswith("co-authored-by")]
detail = "\n".join(lines[:8]) if lines else subject
detail += f"\n(main {sha[:8]}, {len(files)} file{'s' if len(files) != 1 else ''})"
detail = "\n".join(lines[:8]) + "\n" if lines else "" # the board shows the title already
detail += f"(main {sha[:8]}, {len(files)} file{'s' if len(files) != 1 else ''})"
added.append(logbook.add_entry(project, subject, detail, kind=_kind(files), at=_utc(when)))
return added
@@ -88,8 +91,10 @@ def rebuild_stats(project, repo, stats_repo, head):
installer = r"C:\Program Files (x86)\Microsoft Visual Studio\Installer"
if os.path.isdir(installer) and installer.lower() not in env.get("PATH", "").lower():
env["PATH"] = installer + os.pathsep + env.get("PATH", "")
run = lambda *argv: subprocess.run([sys.executable, "-B", *map(str, argv)], cwd=str(stats_repo), env=env,
capture_output=True, text=True, encoding="utf-8", errors="replace", timeout=900)
# below normal priority (cl.exe inherits it), so the lander's and the agents' compiles go first; a timeout ends
# the whole tree, not only python (an orphaned cl.exe went on compiling all of main at normal priority)
run = lambda *argv: lander_gate.run_low([sys.executable, "-B", *map(str, argv)], STATS_TIMEOUT,
cwd=str(stats_repo), env=env, encoding="utf-8", errors="replace")
sync = run(tools / "sync_decomp.py", "--decomp", repo)
if sync.returncode != 0:
raise RuntimeError("sync_decomp: " + (sync.stdout + sync.stderr).strip()[-400:])
@@ -134,12 +139,20 @@ def refresh(project, force=False):
state = _state(project)
if state.get("last_head") == head and not force:
return {"changed": False, "head": head[:8]}
busy = None if force else lander_gate.lander_busy(project)
if busy: # its gates compile all of main: look again next time
return {"changed": False, "head": head[:8], "waiting": busy}
out = {"changed": True, "head": head[:8]}
if state.get("last_head"):
# last_head is where the statistics are; logged_head where the ship's log is (a failed statistics run is retried,
# and must not log the same commits twice)
logged = state.get("logged_head") or state.get("last_head")
if logged and logged != head:
try:
out["entries"] = len(log_commits(project, repo, state["last_head"], head))
out["entries"] = len(log_commits(project, repo, logged, head))
except RuntimeError as error: # e.g. the old commit vanished after a rewrite
out["log_error"] = str(error)
if logged:
state["logged_head"] = head
if stats_repo:
try:
out["stats"] = rebuild_stats(project, snapshot_at(config, repo, head), stats_repo, head)
+21
View File
@@ -38,6 +38,8 @@ review queue, presence and stop flags. Protocol:
10. Before your session ends or your allowance runs out, write handoff(): finished, in flight, next.
11. To read the original machine code use re_disassemble, re_xrefs, re_bytes and re_lookup: they read the project's
own image with the team's current names, so do not write your own disassembler or offset maths.
re_summary(address) gives the overview in one call: name, prototype, size, convention, readable status,
callers and callees with their status, globals, strings, vtable slots and the Mac/Xbox hints.
For Ghidra's view (decompiler, p-code, stack frames, struct field references, types) use ghidra_tools() once,
then ghidra(tool, arguments): one shared read-only Ghidra session holding the team's annotations.
To check a value in the running game (x87 control word, `this`, struct fields), use debugger_tools() and
@@ -563,6 +565,14 @@ def re_lookup(target: str) -> dict:
return _re(lambda r: r.lookup(target))
@mcp.tool
def re_summary(target: str) -> dict:
"""One call before starting on a function: its current name and prototype, size and calling convention (read
from the bytes), whether it is readable on main, its callers and callees with their names and whether each is
readable on main, the globals and strings it uses, the vtable slots it fills, and the Mac/Xbox name hints."""
return _re(lambda r: r.summary(target))
_ghidra = {}
@@ -692,6 +702,17 @@ def ghidra_tools() -> list:
return _gh(lambda g: g.tools())
@mcp.tool()
def ghidra_check() -> dict:
"""Is the shared Ghidra session serving our program? Compares its address range with the project image's base
and size, the SHA-256 of the file it imported with ours, and sample bytes; ok is false when any differs."""
try:
from .reverse import ghidra_check as check
return check(harbor().project.reverse)
except (HarborError, projects.ProjectError) as error:
raise ToolError(str(error)) from None
@mcp.tool()
def ghidra(tool: str, arguments: dict | None = None) -> str:
"""Run one tool in the shared read-only Ghidra session, e.g. ghidra('get_code', {'function': 'GetFogLevel',
+11 -1
View File
@@ -213,6 +213,7 @@ class ProgressMap:
units, unit_index, functions = [], {}, []
game_functions = game_done = 0
tiers = {t: {"functions": 0, "bytes": 0, "game_functions": 0, "game_bytes": 0} for t in STATUSES}
for addr in sorted(facts):
d = facts[addr]
size = int(d.get("size") or 0)
@@ -257,6 +258,11 @@ class ProgressMap:
kind = "game" if counted else "library" if tier == "library" else "engine" if category == "engine" \
else "unsorted" if category == "unsorted" else "library"
u["kinds"][kind] = u["kinds"].get(kind, 0) + size
tiers[tier]["functions"] += 1
tiers[tier]["bytes"] += size
if counted: # the Code bar's game functions, split by tier
tiers[tier]["game_functions"] += 1
tiers[tier]["game_bytes"] += size
if counted:
game_functions += 1
game_done += done
@@ -291,7 +297,11 @@ class ProgressMap:
"source": "logbook symbol table" if symbols else "src headers and names.csv",
"commit": commit, "main": main, "updated_at": stats.get("updated_at"),
"percent": percent, "logbook_percent": stats.get("game_bytes_decompiled_percent"),
"code": code, "data": self._data(facts, rev),
"code": code, "data": self._data(facts, rev), "tiers": tiers,
# every function in the executable (game, engine, platform, CRT), so engine work shows too
"all_code": {"bytes": stats.get("bytes"), "functions": stats.get("functions"),
"decompiled_bytes": ((stats.get("by_status") or {}).get("decompiled") or {}).get("bytes"),
"decompiled_functions": ((stats.get("by_status") or {}).get("decompiled") or {}).get("functions")},
"history": [[p.get("at"), p.get("decompiled"), p.get("commit"), p.get("game_bytes")]
for p in history if p.get("decompiled") is not None],
"statuses": list(STATUSES),
+8
View File
@@ -196,6 +196,14 @@ def _policy(raw, names):
policy["all_per_hour"] = int(policy["all_per_hour"])
except (TypeError, ValueError):
raise ProjectError("policy all_per_hour must be a whole number")
if "submits_per_hour" in policy: # {agent: n}: submissions and resubmissions per hour
caps = policy["submits_per_hour"]
if not isinstance(caps, dict) or not all(k in names for k in caps):
raise ProjectError("policy submits_per_hour must map agents on the roster to whole numbers")
try:
policy["submits_per_hour"] = {k: int(v) for k, v in caps.items()}
except (TypeError, ValueError):
raise ProjectError("policy submits_per_hour must map agents on the roster to whole numbers")
return policy
+4
View File
@@ -207,6 +207,10 @@ class FunctionQueue:
head = self.r.read(addr, 10) if int(facts[addr].get("size") or 0) == 10 else b""
if head[:1] and 0xB8 <= head[0] <= 0xBF and head[5:6] == b"\xe9":
continue # mov reg, imm32; jmp: an at-exit destructor thunk or an SEH handler stub (compiler's)
first = self.r.read(addr, 3)
if len(first) == 3 and first[0] in (0x8B, 0x8D, 0xFF, 0x83, 0xC7, 0x89) and (first[1] & 0xC7) == 0x45 \
and first[2] >= 0x80:
continue # opens on the parent's [ebp-x] with no frame: a catch or unwind funclet (ruling #12777)
score, reasons = self.score(addr, facts, named, rejected, similar_readable, with_convention=False)
ranked.append((score, addr, reasons))
ranked.sort(reverse=True)
+14 -12
View File
@@ -13,6 +13,7 @@ Configured per project in the [logbook] table (main is [logbook] decomp):
behaviour_every_hours = 12
behaviour_window = "01-06" from 01:00 to 06:00 local time; "" for any hour
No run starts while the lander runs its gates ([logbook] lander_lock, harbor.lander_gate).
One run at a time (the server runs one thread for every project), at the lowest process priority, into a .partial file
that replaces the results only once it is complete. The match run holds the same per-checkout lock as re_match_program,
so it waits its turn behind any other whole-program build of main.
@@ -31,7 +32,7 @@ import time
import traceback
from pathlib import Path
from . import logbook
from . import lander_gate, logbook
JOBS = ("match", "behaviour")
RESULTS_KEY = {"match": "match_results", "behaviour": "diffrun_results"}
@@ -180,14 +181,7 @@ def _spawn(argv, cwd, stdout, stderr):
def _kill_tree(pid):
if os.name == "nt":
subprocess.run(["taskkill", "/PID", str(pid), "/T", "/F"], capture_output=True,
creationflags=getattr(subprocess, "CREATE_NO_WINDOW", 0))
else:
try:
os.kill(pid, 9)
except OSError:
pass
lander_gate.kill_tree(pid)
def _alive(pid):
@@ -205,12 +199,12 @@ def _paths(project, job):
return paths
def _argv(project, job, out):
def _argv(project, job, out, decomp=None):
need = configured(project, job)
if job == "match":
script = Path(need["script"])
return [sys.executable, "-B", str(script), "check", "--build", "--decomp",
str(Path(project.logbook["decomp"])), "--json", str(out)], script.parent
str(Path(decomp or project.logbook["decomp"])), "--json", str(out)], script.parent
command = [str(c) for c in need["command"]]
return command + ["--readable", "--jobs", "4", "--json", str(out)], Path(command[-1]).parent
@@ -260,7 +254,12 @@ def _complete(project, state, run, now, returncode=None):
def run_job(project, state, job, head, now, reason=""):
paths = _paths(project, job)
argv, cwd = _argv(project, job, paths["partial"])
decomp = None
if job == "match" and project.logbook.get("results_snapshot") and head:
# a clean checkout of exactly `head`: the live checkout is the lander's and changes under a running build
from .logbook_auto import snapshot_at
decomp = snapshot_at({"snapshot": project.logbook["results_snapshot"]}, project.logbook["decomp"], head)
argv, cwd = _argv(project, job, paths["partial"], decomp)
for stale in [paths["partial"]] + ([paths["text_partial"]] if job == "behaviour" else []):
if stale.exists():
stale.write_bytes(b"") # a leftover of an interrupted run is never taken for this one
@@ -310,6 +309,9 @@ def tick(project, now=None, hour=None):
job, why = choose(settings(project), state, head, now, hour, requested, available)
if not job:
return {"idle": why, "head": head[:8]}
busy = lander_gate.lander_busy(project, now)
if busy: # a request is kept for the next look
return {"waiting": busy, "job": job}
if job == "match":
from .util import HarborError
try:
+230
View File
@@ -7,6 +7,8 @@ Configured per project in a [reverse] table:
imports = 'R:/.../imports.json' optional: [{module, name, slot}]
annotations = 'R:/.../annotations' names.csv, functions.csv, globals.csv, prototypes.h
src = 'R:/.../src' optional: the decompiled C files (<ADDR>_<Name>.c)
sections = 'R:/.../sections.json' optional: [{name, start, end}] (default: sections.json beside the image),
used to find the vtables and pointer tables a function sits in
Every answer comes from those files, read fresh when they change, so the agents see the team's current names.
Nothing here writes anything."""
@@ -42,6 +44,7 @@ class Reverse:
self.conventions_path = Path(config["conventions"]) if config.get("conventions") else None
self.source_files = Path(config["source_files"]) if config.get("source_files") else None
self.mac_names = Path(config["mac_names"]) if config.get("mac_names") else None
self.sections_path = Path(config["sections"]) if config.get("sections") else None
self._cache = {}
# ----------------------------------------------------------------- loading (cached by file mtime)
@@ -491,6 +494,176 @@ class Reverse:
break
return out
# ----------------------------------------------------------------- one-call summary
def _src_index(self):
"""address -> path of src/**/<ADDR>_<Name>.c (one directory walk, reused for 30 seconds)."""
import time
hit = self._cache.get("src_index")
if hit and time.monotonic() - hit[0] < 30:
return hit[1]
out = {}
if self.src and self.src.exists():
for folder, _, files in os.walk(self.src):
for name in files:
m = re.match(r"([0-9A-Fa-f]{6,8})_.*\.c$", name)
if m:
out.setdefault(int(m.group(1), 16), Path(folder) / name)
self._cache["src_index"] = (time.monotonic(), out)
return out
def status_of(self, addr):
"""The decompiled file's status on main ('readable', 'raw', ...), or None when there is no file."""
path = self._src_index().get(addr)
if not path:
return None
try:
with path.open(encoding="utf-8", errors="replace") as f:
head = [next(f, "") for _ in range(8)]
except OSError:
return None
return next((l.split("status:", 1)[1].strip() for l in head if "status:" in l), "raw")
def _sections(self):
"""[(name, start, end)] from sections.json beside the image (or [reverse] sections), else []."""
path = self.sections_path or self.image.with_name("sections.json")
return self._load("sections", path, lambda p: [
(s["name"], _int(s["start"]), _int(s["end"])) for s in json.loads(p.read_text(encoding="utf-8"))] if p else [])
def _code_data_refs(self):
facts = self._facts()
key = ("data_refs", id(facts))
if key not in self._cache:
self._cache[key] = {_int(a) for d in facts.values() for a in d.get("data", [])}
return self._cache[key]
def _mac_vtables(self):
return self._load("mac_vtables", self.mac_names.with_name("vtables.csv") if self.mac_names else None,
lambda p: {int(r["pc_vtable"], 16): r.get("class") for r in
csv.DictReader(p.read_text(encoding="utf-8").splitlines())
if (r.get("pc_vtable") or "").startswith("0x")} if p else {})
def vtable_slots(self, addr):
"""Where the function's address sits in a table of code pointers outside .text: the table start (walked back
over function addresses to the entry the code references), the slot index, and what the table is."""
data = self._bytes()
facts = self._facts()
sections = self._sections()
text = next(((lo, hi) for name, lo, hi in sections if name == ".text"), None)
tables = [(lo, hi) for name, lo, hi in sections if name in (".rdata", ".data")]
if not tables: # no section list: everything past the last function
end = max((a + int(d.get("size") or 0) for a, d in facts.items()), default=self.base)
tables = [(end, self.base + len(data))]
refs, mac = self._code_data_refs(), self._mac_vtables()
is_code = (lambda v: v in facts) if facts else (lambda v: text and text[0] <= v < text[1])
needle = struct.pack("<I", addr)
out = []
for lo, hi in tables:
pos = data.find(needle, max(0, lo - self.base), hi - self.base)
while pos >= 0 and len(out) < 40:
if pos % 4 == 0:
start = pos
while start - 4 >= lo - self.base and (start + self.base) not in refs and \
(start + self.base) not in mac and \
is_code(struct.unpack_from("<I", data, start - 4)[0]) and pos - start < 4 * 400:
start -= 4
table = start + self.base
slot = (pos - start) // 4
name = self.name_of(table)
kind = "vtable" if table in refs or table in mac or (name and "vtable" in name.lower()) \
else "pointer table"
out.append({"table": f"0x{table:X}", "slot": slot, "offset": f"0x{slot * 4:X}", "kind": kind,
**({"table_name": name} if name else {}),
**({"mac_class": mac[table]} if table in mac else {})})
pos = data.find(needle, pos + 1, hi - self.base)
return out
def summary(self, target):
"""Everything a worker needs before starting on a function, in one call: name, prototype, size, convention,
whether it is readable on main, callers and callees (named, with their status on main), globals and strings,
the vtable slots it fills, and the Mac/Xbox name hints."""
addr = self.resolve(target)
func = self.function_at(addr)
if not func:
raise HarborError(f"0x{addr:X} is not inside a known function")
start, size, name = func
d = self._facts()[start]
def fn(a):
status = self.status_of(a)
return {"address": f"0x{a:X}", "name": self.name_of(a), "status": status or "no file",
"readable": status == "readable"}
out = {"address": f"0x{start:X}", "name": name, "size": size, "end": f"0x{start + size:X}"}
if addr != start:
out["asked"] = f"0x{addr:X} (+0x{addr - start:X})"
proto = self._prototypes().get(name) if name else None
out["prototype"] = proto
conv = self.convention(start)
out["convention"] = conv["summary"] if conv else None
if conv:
out["registers_in"], out["ret_pops"] = conv["registers_in"], conv["ret_pops"]
status = self.status_of(start)
out["status"] = status or "no file"
out["readable"] = status == "readable"
path = self._src_index().get(start)
if path and self.src:
out["file"] = str(path.relative_to(self.src.parent)).replace(os.sep, "/")
row = self._csv("names.csv").get(start)
if row and len(row) > 3 and row[3]:
out["note"] = row[3] if len(row[3]) < 600 else row[3][:600] + "..."
callers = sorted(set(self._callers().get(start, [])))
callees = list(dict.fromkeys(_int(a) for a in d.get("calls", [])))
out["callers"] = [fn(a) for a in callers]
out["callees"] = [fn(a) for a in callees]
if d.get("imports"):
out["imports"] = d["imports"]
glob = self._csv("globals.csv")
out["globals"] = [{"address": f"0x{a:X}", "name": self.name_of(a),
**({"type": glob[a][2]} if a in glob and len(glob[a]) > 2 and glob[a][2] else {})}
for a in dict.fromkeys(_int(x) for x in d.get("data", []))]
out["strings"] = d.get("strings", [])
out["vtable_slots"] = self.vtable_slots(start)
out["hints"] = self.origins(start)
out["counts"] = {"callers": len(callers), "callers_readable": sum(c["readable"] for c in out["callers"]),
"callees": len(callees), "callees_readable": sum(c["readable"] for c in out["callees"])}
return out
def format_summary(s):
"""Reverse.summary() as text for the command line."""
mark = lambda f: f"{f['address']} {f['name'] or '(unnamed)'} [{'readable' if f['readable'] else f['status']}]"
lines = [f"{s['name'] or '(unnamed)'} {s['address']}..{s['end']} ({s['size']} bytes)"
+ (f", asked {s['asked']}" if s.get("asked") else ""),
f"status on main: {s['status']}" + (f" ({s['file']})" if s.get("file") else ""),
f"convention: {s['convention'] or 'unknown'}",
"prototype: " + (s["prototype"] or "none in prototypes.h")]
c = s["counts"]
lines.append(f"callers ({c['callers']}, {c['callers_readable']} readable):")
lines += [f" {mark(f)}" for f in s["callers"]] or [" none"]
lines.append(f"callees ({c['callees']}, {c['callees_readable']} readable):")
lines += [f" {mark(f)}" for f in s["callees"]] or [" none"]
if s.get("imports"):
lines.append("imports: " + ", ".join(s["imports"]))
lines.append(f"globals ({len(s['globals'])}):")
lines += [f" {g['address']} {g['name'] or '(unnamed)'}" + (f" : {g['type']}" if g.get("type") else "")
for g in s["globals"]] or [" none"]
if s["strings"]:
lines.append("strings: " + ", ".join(repr(x) for x in s["strings"]))
lines.append("vtable slots:" + ("" if s["vtable_slots"] else " none"))
lines += [f" {v['kind']} {v['table']}" + (f" {v['table_name']}" if v.get("table_name") else "")
+ (f" (Mac class {v['mac_class']})" if v.get("mac_class") else "") + f" slot {v['slot']} (+{v['offset']})"
for v in s["vtable_slots"]]
h = s["hints"]
if h.get("mac_name"):
lines.append(f"Mac name: {h['mac_name'].get('demangled')} ({h['mac_name'].get('confidence')})")
if h.get("source_file"):
sf = h["source_file"]
lines.append(f"Xbox source file: {sf.get('source_file')}" + (f" lines {sf['lines']}" if sf.get("lines") else "")
+ f" ({sf.get('confidence')})")
if not h:
lines.append("Mac/Xbox hints: none")
return "\n".join(lines)
class Ghidra:
"""A small client for the shared, read-only Ghidra MCP server (GhidrAssistMCP over streamable HTTP), so every
@@ -572,6 +745,63 @@ class Ghidra:
return re.sub(r"^\[Context\] Operating on: [^\n]*\n+", "", text)
def _hexdump_bytes(text):
"""The bytes of a GhidrAssistMCP get_data_at hexdump ('401000 53 55 8b ... |SU.|')."""
out = bytearray()
for line in text.splitlines():
m = re.match(r"^\s*[0-9A-Fa-f]{4,16}\s{2}((?:[0-9A-Fa-f]{2}\s{1,2})+)", line + " ")
if m:
out += bytes.fromhex("".join(m.group(1).split()))
return bytes(out)
def ghidra_check(config, samples=16, sample_len=128):
"""Is the shared Ghidra server serving our program? Read-only: compares the program's address range with the
project image (base and size), the SHA-256 of the file Ghidra imported (when that path is readable here) with
the image's, and sample bytes read through Ghidra (get_data_at) with the image at spread-out addresses."""
import hashlib
reverse = Reverse(config)
image = reverse._bytes()
if not image:
raise HarborError(f"the project image {reverse.image} is missing")
base = reverse.base
g = Ghidra(config)
info = g.call("get_binary_info", {})
binaries = g.call("list_binaries", {})
field = lambda label, text: (re.search(rf"^\s*{label}:\s*(.+?)\s*$", text, re.M) or [None, None])[1]
lo, hi = field("Min Address", info), field("Max Address", info)
checks = []
want_lo, want_hi = base, base + len(image) - 1
got = (int(lo, 16), int(hi, 16)) if lo and hi else None
checks.append({"check": "address range", "ok": got == (want_lo, want_hi),
"expected": f"0x{want_lo:X}..0x{want_hi:X} ({len(image)} bytes)",
"ghidra": f"0x{got[0]:X}..0x{got[1]:X} ({got[1] - got[0] + 1} bytes)" if got else info.strip()[:200]})
sha = hashlib.sha256(image).hexdigest()
exe = field("Executable Path", binaries)
local = Path(exe[1:] if exe and re.match(r"^/[A-Za-z]:/", exe) else exe) if exe else None
if local and local.is_file():
theirs = hashlib.sha256(local.read_bytes()).hexdigest()
checks.append({"check": "sha256 of the imported file", "ok": theirs == sha, "expected": sha,
"ghidra": theirs, "file": str(local)})
else:
checks.append({"check": "sha256 of the imported file", "ok": None, "expected": sha,
"ghidra": f"not readable here ({exe or 'no path reported'})"})
step = max(sample_len, (len(image) - sample_len) // max(1, samples - 1)) & ~0xF
bad = []
for i in range(samples):
addr = base + min(i * step, len(image) - sample_len)
theirs = _hexdump_bytes(g.call("get_data_at", {"address": f"0x{addr:X}", "len": sample_len}))
ours = image[addr - base:addr - base + sample_len]
if theirs != ours:
bad.append(f"0x{addr:X}")
checks.append({"check": f"{samples} byte samples of {sample_len}", "ok": not bad,
**({"differ_at": bad} if bad else {})})
ok = all(c["ok"] is not False for c in checks)
return {"ok": ok, "program": field("Name", info), "image": str(reverse.image), "image_sha256": sha,
"image_base": f"0x{base:X}", "image_size": len(image), "functions_in_ghidra": field("Function Count", info),
"checks": checks}
class Debugger:
"""A per-agent x64dbg session through the x64dbg-automate MCP server (stdio), started on first use and kept for
the life of this Harbor process, so breakpoints and the paused game survive between calls. Configured by
+32 -24
View File
@@ -699,7 +699,7 @@ button.chip:hover { color: var(--accent-ink); background: var(--accent-soft); }
<div class="lb-stamp" id="lbStamp">Loading…</div></div></section>
<section class="card" aria-labelledby="h-pm" id="pmCard">
<div class="body">
<h2 class="pm-title" id="h-pm"><span id="pmProject">The project</span> is <b id="pmPct">–</b> decompiled</h2>
<h2 class="pm-title" id="h-pm"><span id="pmProject">The project</span>: game code <b id="pmPct">–</b> decompiled</h2>
<div class="pm-bars" id="pmBars"></div>
<div class="note pm-fresh" id="pmFresh"></div>
<div class="pm-commit"><span id="pmCommit"></span><span class="grow"></span>
@@ -849,7 +849,7 @@ const ic=(n,c)=>'<svg class="i'+(c?' '+c:'')+'" aria-hidden="true"><use href="st
const empty=(icon,title,text)=>'<div class="empty">'+ic(icon)+'<strong>'+esc(title)+'</strong><span>'+(text||'')+'</span></div>';
const INIT={Claude:'CL',Codex:'CX',DeepSeek:'DS',OpenCode:'OC',Kimi:'KM',User:'You'};
const ago=m=>m==null?'never':m<1?'just now':m<120?m+' min ago':m<2880?Math.round(m/60)+' h ago':Math.round(m/1440)+' days ago';
const central=s=>{const d=new Date(s);return isNaN(d)?s:d.toLocaleString('en-US',{timeZone:'America/Chicago',month:'short',day:'numeric',hour:'numeric',minute:'2-digit',timeZoneName:'short'})};
const central=s=>{const d=new Date(s);return isNaN(d)?s:d.toLocaleString('en-US',{timeZone:'America/Chicago',month:'short',day:'numeric',hour:'numeric',minute:'2-digit',hour12:true})+' CST'};
const STATE={submitted:'blue',in_review:'violet',changes_requested:'amber',approved:'green',held:'',landed:'green',withdrawn:'',rejected:'red'};
let data=null,openThread=null,current=null,doctorRan=false,unseen=0;
const HUMAN=()=>data&&data.project?data.project.human:'User';
@@ -1132,17 +1132,21 @@ el('menuBtn').onclick=()=>el('shell').classList.add('open');
el('scrim').onclick=()=>el('shell').classList.remove('open');
window.addEventListener('hashchange',()=>{const v=location.hash.slice(1);if(v&&v!==current)show(v)});
const TIERS=[['exact','Exact match','byte-identical with the game’s own compiler'],['verified','Verified','behaves the same as the game in the emulator'],
['readable','Readable','readable C that passed review'],['deferred','Deferred','set aside with a reason after failed attempts'],['raw','Raw','not decompiled yet'],
['library','Library','C runtime, D3DX9 and other linked libraries: never decompiled'],['engine','Engine','Gamebryo engine code: names only']];
['readable','Readable','readable C that passed review'],['deferred','Deferred','set aside with a reason after failed attempts'],['raw','Raw','game code not decompiled yet'],['unsorted','Unsorted','not classified yet, outside the known library ranges (the few inside them count as Library)'],
['library','Library','C runtime, D3DX9 and other linked libraries: never decompiled'],['engine','Engine','Gamebryo engine code not decompiled: names only']];
const fmt=n=>Number(n||0).toLocaleString('en-US');
function renderDecomp(){renderProgress();renderNextUp();renderDeferred()}
async function renderProgress(){let p;try{p=await get('api/decomp/progress')}catch(e){put('progress','<div class="empty err">'+esc(e.message)+'</div>');return}
const t=p.tiers,own=['exact','verified','readable','deferred','raw'].reduce((a,k)=>a+t[k].functions,0),done=t.exact.functions+t.verified.functions+t.readable.functions;
el('progressSum').textContent=fmt(done)+' of '+fmt(own)+' game functions decompiled';
async function renderProgress(){let p;try{await pmLoad();p=PM.doc&&PM.doc.tiers?PM.doc:await get('api/decomp/progress')}catch(e){put('progress','<div class="empty err">'+esc(e.message)+'</div>');return}
const t=tz(p.tiers),g=k=>t[k].game_functions!=null?t[k].game_functions:t[k].functions,
own=['exact','verified','readable','deferred','raw'].reduce((a,k)=>a+g(k),0),done=g('exact')+g('verified')+g('readable');
el('progressSum').textContent=fmt(done)+' of '+fmt(own)+' game functions decompiled · the bar shows every function';
put('progress',tierBar(t)+'<div class="lb-legend">'+TIERS.map(([k,l])=>'<span><i class="tier-key tier-'+k+'"></i>'+esc(l)+' '+fmt(t[k].functions)+'</span>').join('')+'</div><div class="note" style="margin-top:10px">Bytes, categories, the trend and the ship’s log: <a href="#logbook" data-go="logbook">open the Logbook</a>.</div>');}
function tierTable(t){return '<table style="margin-top:14px"><thead><tr><th>Tier</th><th style="text-align:right">Functions</th><th style="text-align:right">Bytes</th><th>Meaning</th></tr></thead><tbody>'
+TIERS.map(([k,l,d])=>'<tr><td><span class="tier-key tier-'+k+'"></span>'+esc(l)+'</td><td class="num">'+fmt(t[k].functions)+'</td><td class="num">'+fmt(t[k].bytes)+'</td><td class="note">'+esc(d)+'</td></tr>').join('')+'</tbody></table>'}
function tierBar(t){return '<div class="tierbar" role="img" aria-label="'+esc(TIERS.map(([k,l])=>l+' '+t[k].functions).join(', '))+'">'+TIERS.map(([k,l])=>t[k].functions?'<span class="tier-'+k+'" style="flex:'+t[k].functions+'" title="'+esc(l)+': '+fmt(t[k].functions)+' functions"></span>':'').join('')+'</div>'}
const tz=t=>Object.fromEntries(TIERS.map(([k])=>[k,t[k]||{functions:0,bytes:0}]));
function tierTable(t){t=tz(t);const g=k=>t[k].game_functions,has=g('raw')!=null,sum=f=>TIERS.reduce((a,[k])=>a+(f(k)||0),0);
return '<table style="margin-top:14px"><thead><tr><th>Tier</th>'+(has?'<th style="text-align:right">Game functions</th>':'')+'<th style="text-align:right">All functions</th><th style="text-align:right">Bytes (all)</th><th>Meaning</th></tr></thead><tbody>'
+TIERS.map(([k,l,d])=>'<tr><td><span class="tier-key tier-'+k+'"></span>'+esc(l)+'</td>'+(has?'<td class="num">'+fmt(g(k))+'</td>':'')+'<td class="num">'+fmt(t[k].functions)+'</td><td class="num">'+fmt(t[k].bytes)+'</td><td class="note">'+esc(d)+'</td></tr>').join('')
+'<tr><td><b>Total</b></td>'+(has?'<td class="num"><b>'+fmt(sum(g))+'</b></td>':'')+'<td class="num"><b>'+fmt(sum(k=>t[k].functions))+'</b></td><td class="num"><b>'+fmt(sum(k=>t[k].bytes))+'</b></td><td class="note">exact + verified + readable = decompiled</td></tr></tbody></table>'}
function tierBar(t){t=tz(t);return '<div class="tierbar" role="img" aria-label="'+esc(TIERS.map(([k,l])=>l+' '+t[k].functions).join(', '))+'">'+TIERS.map(([k,l])=>t[k].functions?'<span class="tier-'+k+'" style="flex:'+t[k].functions+'" title="'+esc(l)+': '+fmt(t[k].functions)+' functions"></span>':'').join('')+'</div>'}
async function renderNextUp(){if(!el('nextUp')._html)put('nextUp',empty('clock','Ranking the bands…','This takes a few seconds the first time.'));let q;
try{q=await get('api/decomp/queues')}catch(e){put('nextUp','<div class="empty err">'+esc(e.message)+'</div>');return}
put('nextUp',q.length?q.map(a=>'<div class="band"><div class="cell-who">'+avatar(a.agent,'xs')+'<b>'+esc(a.agent)+'</b><span class="note mono">'+esc(a.bands.map(b=>b.join('–')).join(', '))+'</span></div>'
@@ -1178,24 +1182,27 @@ const LB_CATS=[['game','Game code','Firaxis game logic: decompiled to readable C
['platform','Platform','Miles, Bink, D3D9 and input wrappers: replaced in the port'],['crt','C runtime','MSVC 7.1 runtime: replaced by Rust’s standard library'],['unsorted','Unsorted','not classified yet; each lands in one of the four above']];
let LB=null;
const pct=n=>Number(n||0).toFixed(Number(n)>=10?1:2)+'%';
const kb=b=>b>=1048576?(b/1048576).toFixed(2)+' MB':Math.round((b||0)/1024)+' KB';
const kb=b=>b>=1e6?(b/1e6).toFixed(2)+' MB':Math.round((b||0)/1e3)+' kB';
const cssVar=n=>getComputedStyle(el('view-logbook')).getPropertyValue(n).trim();
function renderLbFigs(){const s=(LB&&LB.stats)||{},cat=k=>(s.by_category||{})[k]||{functions:0,bytes:0};
/* by_status.decompiled counts every category (engine and platform too); the game-only count comes from the progress map */
const total=s.functions||0,game=cat('game').functions,uns=cat('unsorted').functions,all=(s.by_status&&s.by_status.decompiled||{}).functions||0;
const c=PM.doc&&PM.doc.code,done=c?c.decompiled_functions:null,handled=done==null?null:total-uns-Math.max(0,game-done);
put('lbFigs',[[handled==null?'…':pct(total?100*handled/total:0)+' done',handled==null?'counting…':fmt(handled)+' of '+fmt(total)+' functions handled','game code decompiled, everything else identified'],
[done==null?'…':fmt(done)+' of '+fmt(game),'game functions decompiled to readable C',done==null?fmt(all)+' decompiled across all code':fmt(game-done)+' to go · '+fmt(all)+' decompiled across all code'],
[fmt(uns),'functions not sorted yet','each one is either game code (to decompile) or engine/library code (to name)']]
.map(([v,l,n])=>'<div class="lb-fig"><div class="v">'+esc(v)+'</div><div class="l">'+esc(l)+'</div><div class="n">'+esc(n)+'</div></div>').join(''))}
async function renderLogbook(){try{LB=await get('api/logbook')}catch(e){put('lbStamp','<span class="err">'+esc(e.message)+'</span>');return}
const s=LB.stats||{},cat=k=>(s.by_category||{})[k]||{functions:0,bytes:0};
const cur=LB.milestones.find(m=>/progress/i.test(m.state||''));el('lbMilestone').textContent=cur?'Milestone '+cur.code+' · '+cur.title:'Milestones';
const cur=LB.milestones.filter(m=>/progress/i.test(m.state||'')).pop();el('lbMilestone').textContent=cur?'Milestone '+cur.code+' · '+cur.title:'Milestones';
const au=LB.auto||{};put('lbStamp',(s.updated_at?'Statistics updated <b>'+esc(central(s.updated_at))+'</b><br>main at <b>'+esc(s.commit||'–')+'</b>':'No statistics yet')+(au.last_refresh?'<br>Keeps itself current: last checked '+esc(central(au.last_refresh))+(au.last_error?' · <span class="err">'+esc(String(au.last_error).slice(0,120))+'</span>':''):''));
const total=s.functions||0,game=cat('game').functions,done=(s.by_status&&s.by_status.decompiled||{}).functions||0,uns=cat('unsorted').functions;
const handled=total-uns-Math.max(0,game-done);
put('lbFigs',[[pct(total?100*handled/total:0)+' done',fmt(handled)+' of '+fmt(total)+' functions handled','game code decompiled, everything else identified'],
[fmt(done)+' of '+fmt(game),'game functions decompiled to readable C',fmt(game-done)+' to go'],
[fmt(uns),'functions not sorted yet','each one is either game code (to decompile) or engine/library code (to name)']]
.map(([v,l,n])=>'<div class="lb-fig"><div class="v">'+esc(v)+'</div><div class="l">'+esc(l)+'</div><div class="n">'+esc(n)+'</div></div>').join(''));
renderLbFigs();
put('lbLegend',LB_CATS.map(([k,n])=>'<span><i class="lb-sw'+(k==='unsorted'?' lb-hatch':'')+'" style="background:var(--lb-'+k+')"></i>'+esc(n)+'</span>').join('')+'<span><i class="lb-sw" style="background:var(--lb-done)"></i>decompiled share under each column</span>');
drawLbMap();pmLoad().then(()=>{if(PM.doc)pmLayout()});renderLbWhere(s);renderLbTrend(LB.history);renderLbMilestones(LB.milestones);
drawLbMap();pmLoad().then(()=>{if(PM.doc){pmLayout();renderLbFigs();renderLbWhere(s)}});renderLbWhere(s);renderLbTrend(LB.history);renderLbMilestones(LB.milestones);
el('lbLogCount').textContent=LB.log_total?LB.log_total+' entries':'';
put('lbLog',LB.log.length?LB.log.map(e=>'<li><time>'+esc(central(e.at))+'</time><div><div class="t"><span class="kind">'+esc(e.kind)+'</span>'+esc(e.title)+'</div><div class="d">'+esc(e.detail)+'</div></div></li>').join(''):'<li class="note">No entries yet.</li>')}
put('lbLog',LB.log.length?LB.log.map(e=>'<li><time>'+esc(central(e.at))+'</time><div><div class="t"><span class="kind">'+esc(e.kind)+'</span>'+esc(e.title)+'</div><div class="d">'+esc(String(e.detail||'').startsWith(e.title)?String(e.detail).slice(e.title.length).replace(/^\n/,''):e.detail)+'</div></div></li>').join(''):'<li class="note">No entries yet.</li>')}
async function renderLbWhere(s){const total=s.bytes||0,cat=k=>(s.by_category||{})[k]||{functions:0,bytes:0},uns=cat('unsorted'),sorted=total?100*(total-uns.bytes)/total:0;
let tiers='';try{const p=await get('api/decomp/progress');tiers='<h3 class="sub-h" style="margin-top:18px">Game functions by how far they have come</h3>'+tierBar(p.tiers)+tierTable(p.tiers)}catch(e){}
let tiers='';try{const p=PM.doc&&PM.doc.tiers?PM.doc:await get('api/decomp/progress');tiers='<h3 class="sub-h" style="margin-top:18px">Every function by how far it has come</h3><div class="note">Live from main. The game column is the one the headline counts; engine and platform functions that were decompiled anyway sit in the same tiers.</div>'+tierBar(p.tiers)+tierTable(p.tiers)}catch(e){}
put('lbWhere','<div><span class="lb-big">'+pct(sorted)+'</span> <span class="note">of the executable’s code is sorted into a category · '+fmt(uns.functions)+' functions ('+kb(uns.bytes)+') still unsorted</span></div>'
+'<div class="lb-stack" role="img" aria-label="Code bytes by category">'+LB_CATS.map(([k,n])=>{const sh=total?100*cat(k).bytes/total:0;return '<i class="'+(k==='unsorted'?'lb-hatch':'')+'" style="flex:0 0 '+sh.toFixed(3)+'%;background:var(--lb-'+k+')" title="'+esc(n)+': '+sh.toFixed(1)+'% of bytes"></i>'}).join('')+'</div>'
+'<div class="lb-axis"><span>0</span><span>sorted '+pct(sorted)+' · unsorted '+pct(100-sorted)+'</span><span>'+kb(total)+'</span></div>'
@@ -1227,7 +1234,8 @@ const kB=b=>b>=1e6?(b/1e6).toFixed(2)+' MB':(b/1e3).toFixed(2)+' kB';
const hx=a=>'0x'+Number(a).toString(16).toUpperCase();
const pc=(b,t)=>(t?100*b/t:0).toFixed(2);
function rel(iso){const t=Date.parse(iso||'');if(!t)return 'at an unknown time';const s=(Date.now()-t)/1000;if(s<90)return 'just now';const m=s/60;if(m<90)return Math.round(m)+' minutes ago';const h=m/60;if(h<36)return Math.round(h)+' hours ago';return Math.round(h/24)+' days ago'}
async function pmLoad(){if(PM.busy)return;PM.busy=true;try{
function pmLoad(){if(!PM.pending)PM.pending=pmLoadOnce().finally(()=>{PM.pending=null});return PM.pending}
async function pmLoadOnce(){if(PM.busy)return;PM.busy=true;try{
const d=await get('api/logbook/progress-map'+(PM.version?'?have='+encodeURIComponent(PM.version):''));if(d.same)return;
const U=d.unit_fields;d.units=d.units.map((u,i)=>{const o={i,fns:[]};U.forEach((k,j)=>o[k]=u[j]);o.pct=o.counted?100*(o.exact+o.verified+o.readable)/o.counted:null;o.lname=o.name.toLowerCase();return o});
d.functions.forEach(f=>{const o={unit:f[0],address:f[1],size:f[2],status:d.statuses[f[3]],name:f[4]};o.lname=o.name.toLowerCase();d.units[o.unit].fns.push(o)});delete d.functions;
@@ -1254,7 +1262,7 @@ function pmHeader(){const d=PM.doc;if(!d)return;const h=d.history,pt=PM.hist!=nu
el('pmProject').textContent=d.project;put('pmFresh',pt?'':pmFresh(d.results));
if(!pt){el('pmPct').textContent=d.percent.toFixed(2)+'%';
put('pmBars',pmBar('Code',kB(c.bytes),pmCodeSegs(c,c.bytes),c.bytes,'game code · '+fmt(c.decompiled_functions)+' of '+fmt(c.functions)+' functions')
+(d.data?pmBar('Data',fmt(d.data.addresses)+' addresses',[['pm-named',d.data.named,pc(d.data.named,d.data.addresses)+'% of the data addresses the code touches have a name in globals.csv ('+fmt(d.data.named)+' of '+fmt(d.data.addresses)+')']],d.data.addresses,'globals named'):''));
+(d.all_code&&d.all_code.bytes?pmBar('All code',kB(d.all_code.bytes),[['pm-readable',d.all_code.decompiled_bytes||0,pc(d.all_code.decompiled_bytes||0,d.all_code.bytes)+'% of the whole executable decompiled ('+fmt(d.all_code.decompiled_functions||0)+' of '+fmt(d.all_code.functions)+' functions)']],d.all_code.bytes,'game + engine + platform · '+fmt(d.all_code.decompiled_functions||0)+' of '+fmt(d.all_code.functions)+' functions'):'')+(d.data?pmBar('Data',fmt(d.data.addresses)+' addresses',[['pm-named',d.data.named,pc(d.data.named,d.data.addresses)+'% of the data addresses the code touches have a name in globals.csv ('+fmt(d.data.named)+' of '+fmt(d.data.addresses)+')']],d.data.addresses,'globals named'):''));
const at=d.commit.at||d.updated_at,moved=d.main.sha&&d.commit.sha&&d.main.sha.slice(0,7)!==String(d.commit.sha).slice(0,7);
put('pmCommit','Updated <b title="'+esc(at||'')+'">'+esc(rel(at))+'</b> in commit <b class="mono">'+esc(d.commit.sha||'–')+'</b>'+(moved?' <span class="note">· main has moved on to '+esc(d.main.sha)+'; the statistics follow within minutes</span>':''));
el('pmHistNote').textContent=d.percent!==d.logbook_percent&&d.logbook_percent!=null?'The Logbook statistics say '+d.logbook_percent+'%; they are being rebuilt.':''}
@@ -1360,7 +1368,7 @@ const fmt=s=>esc(s).replace(/`([^`]+)`/g,'<code>$1</code>');
const DAY_FMT=new Intl.DateTimeFormat('en-US',{timeZone:CENTRAL,year:'numeric',month:'numeric',day:'numeric'});
const TIME_FMT=new Intl.DateTimeFormat('en-US',{timeZone:CENTRAL,hour:'numeric',minute:'2-digit',timeZoneName:'short'});
const centralDay=d=>DAY_FMT.format(d);
const centralTime=d=>TIME_FMT.format(d);
const centralTime=d=>TIME_FMT.format(d).replace(/ C[DS]T$/,'')+' CST';
const since=s=>{if(!s)return 'never';const m=Math.round((Date.now()-new Date(s))/60000);return m<1?'just now':m<120?m+' min ago':m<2880?Math.round(m/60)+' h ago':Math.round(m/1440)+' days ago'};
let people=[],byName={},human='User',messages=[],lastId=0,filter='All',thread=null,sending=false,receipts={},overviewData=null,loaded=false;
async function getJSON(path){const r=await fetch(path,{cache:'no-store'});const j=await r.json().catch(()=>({}));if(!r.ok)throw new Error(j.error||('HTTP '+r.status));return j}
+48
View File
@@ -58,6 +58,54 @@ class LogbookAuto(HarborCase):
self.assertTrue(entry["at"].endswith("Z"))
self.assertFalse(logbook_auto.refresh(self.h.project)["changed"])
def test_waits_for_the_lander_and_a_failed_stats_run_logs_nothing_twice(self):
import os
import time
from unittest import mock
logbook_auto._save_state(self.h.project, {"last_head": self.first})
lock = self.dir / "lander-busy.lock"
lock.write_text("x", encoding="utf-8")
self.h.project.logbook.update(lander_lock=str(lock), stats_repo=str(self.dir / "stats"))
out = logbook_auto.refresh(self.h.project)
self.assertFalse(out["changed"])
self.assertIn("lander", out["waiting"])
self.assertEqual(logbook.read_all(self.h.project)["log"], [])
old = time.time() - 61 * 60
os.utime(lock, (old, old)) # stale: ignored
with mock.patch.object(logbook_auto, "rebuild_stats", side_effect=RuntimeError("timed out")):
out = logbook_auto.refresh(self.h.project)
self.assertEqual((out["entries"], out["stats_error"]), (1, "timed out"))
again = logbook_auto.refresh(self.h.project) # the statistics are retried ...
self.assertIn("stats_error", again)
self.assertNotIn("entries", again) # ... but the commit is not relogged
self.assertEqual(len(logbook.read_all(self.h.project)["log"]), 1)
class LanderGate(unittest.TestCase):
def test_run_low_lowers_priority_and_ends_the_tree_on_timeout(self):
import os
import sys
import time
from harbor import lander_gate
script = "import subprocess, sys, time; subprocess.Popen([sys.executable, '-c', 'import time; time.sleep(60)']);" \
" print('ok', flush=True); time.sleep(60)"
from unittest import mock
seen = {}
real = subprocess.Popen
def popen(argv, **kw):
seen.update(kw)
return real([sys.executable, "-c", "pass"], stdout=subprocess.PIPE, stderr=subprocess.PIPE, text=True)
with mock.patch.object(lander_gate.subprocess, "Popen", popen):
self.assertEqual(lander_gate.run_low(["x"], 60).returncode, 0)
if os.name == "nt":
self.assertTrue(seen["creationflags"] & subprocess.BELOW_NORMAL_PRIORITY_CLASS)
start = time.monotonic()
with self.assertRaises(subprocess.TimeoutExpired):
lander_gate.run_low([sys.executable, "-c", script], 3)
self.assertLess(time.monotonic() - start, 45) # the grandchild did not hold the pipes
class ProgressMapTests(HarborCase):
"""GET /api/logbook/progress-map: units, tiers and the headline, by the Logbook's own definitions."""
+13
View File
@@ -208,6 +208,19 @@ class Refresher(HarborCase):
rr.save_state(self.h.project, st)
self.assertIsNone(rr.freshness(self.h.project)["match"]["sha"])
def test_waits_while_the_lander_runs_its_gates(self):
lock = self.dir / "lander-busy.lock"
self.h.project.logbook["lander_lock"] = str(lock)
lock.write_text("lander", encoding="utf-8")
os.utime(lock, (T0 - 300, T0 - 300))
rr.request(self.h.project, "match")
out = rr.tick(self.h.project, now=T0, hour=12)
self.assertIn("lander", out["waiting"])
self.assertEqual(self.calls, [])
self.assertEqual(rr._requested(self.h.project), {"match"}) # kept for the next look
self.assertTrue(rr.tick(self.h.project, now=T0 + 61 * 60, hour=12)["ok"]) # a lock over an hour old is stale
self.assertEqual(len(self.calls), 1)
def test_runs_at_idle_priority(self):
seen = {}
+99
View File
@@ -58,6 +58,105 @@ class ReverseTools(unittest.TestCase):
with self.assertRaises(HarborError):
Reverse({})
def test_summary_names_status_vtables_and_hints(self):
image = bytearray((self.d / "image.bin").read_bytes())
image[0xC0:0xC8] = struct.pack("<II", 0x401000, 0x401040) # 0x4010C0: a two-slot vtable
(self.d / "image.bin").write_bytes(bytes(image))
(self.d / "sections.json").write_text(json.dumps([
{"name": ".text", "start": 0x401000, "end": 0x401080}, {"name": ".rdata", "start": 0x4010B0,
"end": 0x401100}]), encoding="utf-8")
facts = [json.loads(l) for l in (self.d / "facts.jsonl").read_text(encoding="utf-8").splitlines()]
facts[1]["data"] = ["0x4010C0"] # the constructor installs it
facts[0]["strings"] = [".cue"]
(self.d / "facts.jsonl").write_text("\n".join(json.dumps(f) for f in facts), encoding="utf-8")
src = self.d / "src" / "game"
src.mkdir(parents=True)
(src / "401040_Helper.c").write_text("/* Helper\n * status: readable\n */\nvoid Helper(void) {}\n",
encoding="utf-8")
(self.d / "mac.csv").write_text("pc_address,demangled,confidence\n0x00401000,Level::Get(),high\n",
encoding="utf-8")
r = Reverse({"image": str(self.d / "image.bin"), "base": "0x401000", "facts": str(self.d / "facts.jsonl"),
"annotations": str(self.d / "annotations"), "src": str(self.d / "src"),
"mac_names": str(self.d / "mac.csv")})
s = r.summary("GetLevel")
self.assertEqual((s["name"], s["size"], s["status"], s["readable"]), ("GetLevel", 18, "no file", False))
self.assertIn("plain ret", s["prototype"])
self.assertEqual(s["callees"], [{"address": "0x401040", "name": "Helper", "status": "readable",
"readable": True}])
self.assertEqual(s["globals"], [{"address": "0x401080", "name": "g_root", "type": "int"}])
self.assertEqual(s["strings"], [".cue"])
self.assertEqual(s["vtable_slots"], [{"table": "0x4010C0", "slot": 0, "offset": "0x0", "kind": "vtable"}])
self.assertEqual(s["hints"]["mac_name"]["demangled"], "Level::Get()")
h = r.summary("0x401040")
self.assertTrue(h["readable"])
self.assertEqual([c["name"] for c in h["callers"]], ["GetLevel"])
self.assertEqual(h["vtable_slots"][0]["slot"], 1)
from harbor.reverse import format_summary
self.assertIn("vtable 0x4010C0 slot 1 (+0x4)", format_summary(h))
class GhidraCheck(unittest.TestCase):
"""ghidra_check compares the served program with the project image: range, SHA-256 of the file, byte samples."""
def serve(self, lo, hi, path, image):
import http.server
import threading
class Fake(http.server.BaseHTTPRequestHandler):
def log_message(self, *a):
pass
def do_POST(self):
body = json.loads(self.rfile.read(int(self.headers["Content-Length"])))
if "id" not in body:
self.send_response(202)
self.end_headers()
return
params = body.get("params", {})
name, args = params.get("name"), params.get("arguments", {})
text = ""
if name == "get_binary_info":
text = f"Program Information:\nName: image.bin\nImage Base: 00000000\nMin Address: {lo:08x}\n" \
f"Max Address: {hi:08x}\nFunction Count: 2\n"
elif name == "list_binaries":
text = f"Open Programs in Ghidra:\n\n1. image.bin [ACTIVE]\n Executable Path: /{path}\n"
elif name == "get_data_at":
a, n = int(args["address"], 16), int(args["len"])
chunk = image[a - 0x401000:a - 0x401000 + n]
rows = [f"{a + i:x} " + " ".join(f"{b:02x}" for b in chunk[i:i + 16]) + " |....|"
for i in range(0, len(chunk), 16)]
text = f"Hexdump at {a:08x} ({n} bytes):\n\n" + "\n".join(rows) + "\n"
result = {"content": [{"type": "text", "text": text}]} if body["method"] == "tools/call" else {}
data = json.dumps({"jsonrpc": "2.0", "id": body["id"], "result": result}).encode()
self.send_response(200)
self.send_header("Content-Type", "application/json")
self.send_header("Content-Length", str(len(data)))
self.end_headers()
self.wfile.write(data)
server = http.server.ThreadingHTTPServer(("127.0.0.1", 0), Fake)
threading.Thread(target=server.serve_forever, daemon=True).start()
self.addCleanup(server.server_close)
self.addCleanup(server.shutdown)
return f"http://127.0.0.1:{server.server_port}/mcp"
def test_match_and_mismatch(self):
from harbor.reverse import ghidra_check
d = temp_dir()
self.addCleanup(shutil.rmtree, d, True)
image = bytes(range(256)) * 4
(d / "image.bin").write_bytes(image)
config = {"image": str(d / "image.bin"), "base": "0x401000"}
good = ghidra_check(config | {"ghidra_url": self.serve(0x401000, 0x4013FF, (d / "image.bin").as_posix(), image)})
self.assertTrue(good["ok"], good)
self.assertEqual([c["ok"] for c in good["checks"]], [True, True, True])
other = d / "other.bin"
other.write_bytes(image[:-1] + b"\x00")
bad = ghidra_check(config | {"ghidra_url": self.serve(0x401000, 0x4013FE, other.as_posix(),
image[:512] + b"\0" * 512)})
self.assertFalse(bad["ok"])
self.assertEqual([c["ok"] for c in bad["checks"]], [False, False, False])
class GhidraForwarder(unittest.TestCase):
"""The ghidra() tool against a fake streamable-HTTP MCP server: a session, a polled async task, a refusal."""
+15 -2
View File
@@ -18,6 +18,19 @@ sys.path.insert(0, str(Path(__file__).resolve().parent.parent))
from harbor import projects # noqa: E402
def cst(iso):
"""'2026-10-07T23:21:05Z' -> '18:21 CST' (US Central, as the User reads every time)."""
import datetime
from zoneinfo import ZoneInfo
try:
when = datetime.datetime.fromisoformat(str(iso).replace("Z", "+00:00"))
except ValueError:
return str(iso)
if when.tzinfo is None:
when = when.replace(tzinfo=datetime.timezone.utc)
return when.astimezone(ZoneInfo("America/Chicago")).strftime("%H:%M") + " CST"
def snapshot(conn, me):
q = lambda sql, *a: conn.execute(sql, a).fetchall()
return {
@@ -88,7 +101,7 @@ def main():
for agent, last in stalled(conn, project, args.me, args.stall):
if agent in {a.strip() for a in args.ignore.split(",") if a.strip()}:
continue
found.append(f"stalled: {agent} has made no tool call since {last[11:16]}Z (not stopped) - wake it")
found.append(f"stalled: {agent} has made no tool call since {cst(last)} (not stopped) - wake it")
if found:
print("\n".join(found))
return 0
@@ -96,7 +109,7 @@ def main():
tasks = conn.execute("SELECT state, COUNT(*) FROM tasks GROUP BY state").fetchall()
last = conn.execute("SELECT agent, MAX(at) FROM activity_log GROUP BY agent").fetchall()
print(f"quiet for {args.minutes:g} min; open review items {open_items}; tasks {dict(tasks)}; last tool call "
+ ", ".join(f"{a} {t[11:16]}Z" for a, t in last))
+ ", ".join(f"{a} {cst(t)}" for a, t in last))
return 0