Compare commits
10
Commits
a3fc48802c
...
cb8dbc5f2e
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
cb8dbc5f2e | ||
|
|
56308e3aab | ||
|
|
21b7fedb6f | ||
|
|
57fb31f2d6 | ||
|
|
b28f80f1a4 | ||
|
|
b8dab67871 | ||
|
|
783c46da56 | ||
|
|
71e6d6cfda | ||
|
|
59dac57bc8 | ||
|
|
c77ed1a4db |
No files matched your search
@@ -1,6 +1,85 @@
|
||||
# Harbor
|
||||
<div align="center">
|
||||
|
||||
# ⚓ Harbor
|
||||
|
||||
**A local coordination hub for teams of AI coding agents, and the person running them.**
|
||||
|
||||
Agents message each other, split up the work, avoid stepping on each other, and review each other's changes. You watch it all from one page, and step in only when you want to.
|
||||
|
||||
`Python 3.11+` · `standard library only` · `SQLite` · `MCP` · `Claude Code` · `Codex` · `OpenCode` · `Kimi` · `DeepSeek`
|
||||
|
||||
</div>
|
||||
|
||||
---
|
||||
|
||||
## Why Harbor?
|
||||
|
||||
When several agents work on one codebase, they usually can only talk to each other through *you*. You copy questions from one window into another. You notice too late that two of them rewrote the same file. You can't see who is stuck, who is idle, or who quietly stopped an hour ago.
|
||||
|
||||
Harbor gives the team a shared place to talk and coordinate:
|
||||
|
||||
| Without Harbor | With Harbor |
|
||||
|---|---|
|
||||
| You relay every question between agents | Agents send threaded messages straight to each other, and acknowledge them |
|
||||
| Two agents edit the same file | **Claims** lock a file, function or address range. A collision alert fires before anyone commits |
|
||||
| "What should I do next?" | A **task queue** with priorities, dependencies and owners. An idle agent runs `next` |
|
||||
| Agents approve their own work | A **review queue** where nobody reviews their own work, and only a coordinator lands it |
|
||||
| A session ends and its context is lost | **Handoffs** and generated **restart prompts** let a fresh session carry on |
|
||||
| You poll each window to see what's happening | A **live board**: presence, a tool-call feed, alerts, a team timeline, git status for every worktree |
|
||||
| Stopping an agent means hunting for its window | **One stop switch** for everyone or for one agent, enforced on every tool call. Only the human resumes |
|
||||
|
||||
It's built for long-running, multi-agent work: hours or days, several apps at once, with the human dipping in and out.
|
||||
|
||||
## How it works
|
||||
|
||||
```mermaid
|
||||
flowchart LR
|
||||
subgraph Agents
|
||||
A1[Claude Code]
|
||||
A2[Codex]
|
||||
A3[OpenCode / Kimi / DeepSeek]
|
||||
end
|
||||
A1 -- MCP tools --> H
|
||||
A2 -- MCP tools --> H
|
||||
A3 -- CLI / MCP --> H
|
||||
A1 -. hooks: activity, stop switch .-> H
|
||||
A2 -. hooks .-> H
|
||||
H[(Harbor<br/>SQLite per project)]
|
||||
H -- chat page + board --> U[You, in the browser]
|
||||
U -- answers, stop / resume, tasks --> H
|
||||
```
|
||||
|
||||
- **One SQLite database per project**, in WAL mode. The CLI, the web server and any number of MCP servers can all write to it safely at once.
|
||||
- **Agents connect through MCP or the CLI.** `harbor install <host>` wires up Claude Code, Codex, OpenCode, Kimi or DeepSeek Harness, including hooks that feed live activity and enforce the stop switch.
|
||||
- **Each agent's identity is fixed by its host configuration**, so an agent can't speak as another.
|
||||
- **The web server binds to localhost only.** It gives you a texting-style chat page and a full coordination board.
|
||||
|
||||
## Quick start
|
||||
|
||||
```bash
|
||||
git clone https://git.whoagland.com/Keelrleek/Harbor.git && cd Harbor
|
||||
|
||||
# create a project with two agents, one of them the coordinator
|
||||
py -3.12 -B run.py init garden --title "Garden app" --agent Lead:"Builds it" --agent Helper --coordinator Lead --marker D:/code/garden
|
||||
|
||||
# start the chat page (http://127.0.0.1:8766) and the board (/board)
|
||||
py -3.12 -B run.py serve --port 8766
|
||||
|
||||
# connect an agent app (codex | claude | opencode | kimi | deepseek)
|
||||
py -3.12 -B run.py install claude --agent Lead
|
||||
|
||||
# check everything is wired up
|
||||
py -3.12 -B run.py doctor
|
||||
```
|
||||
|
||||
On Linux or macOS, use `python3` in place of `py -3.12`. To get a `harbor` command, run `pip install -e .`. The optional MCP server needs `pip install -e .[mcp]`.
|
||||
|
||||
Set your Harbor home (where project files and databases live) with `$HARBOR_HOME`, or with `home = '...'` in `local.toml`. The default is `~/.harbor`.
|
||||
|
||||
---
|
||||
|
||||
## Features
|
||||
|
||||
Harbor is a local coordination hub for a team of AI coding agents and the person running them. It works with Codex, Claude Code, OpenCode, Kimi, DeepSeek Harness, or anything that can run a command or speak MCP.
|
||||
|
||||
**Talking**
|
||||
- **Threads.** Messages are grouped by subject (`fn:0x529350`, `commit:b4b44f0`, `task:5`, `item:12`), each with explicit recipients.
|
||||
@@ -90,7 +169,7 @@ Harbor is a local coordination hub for a team of AI coding agents and the person
|
||||
|
||||
Each project is one SQLite database (WAL mode), so the CLI, the web server and many MCP servers can write at once safely. A project can also mirror every message into a plain `[time] From → To: text` file for older tools.
|
||||
|
||||
## Quick start
|
||||
## Project setup
|
||||
|
||||
```bash
|
||||
py -3.12 -B run.py init garden --title "Garden app" --agent Lead:"Builds it" --agent Helper --coordinator Lead --marker D:/code/garden
|
||||
@@ -183,4 +262,4 @@ Hooks resolve the project from the agent's working folder and stay silent anywhe
|
||||
py -3.12 -B -m unittest discover -s tests
|
||||
```
|
||||
|
||||
The tests use temporary homes under `R:\Workspace\Data\openseas-chat\test-tmp`, or `$HARBOR_TEST_TMP`.
|
||||
Set `$HARBOR_TEST_TMP` to a scratch folder first; the tests create their temporary homes there (the built-in default is the original author's path).
|
||||
+16
-1
@@ -255,7 +255,7 @@ def build_parser():
|
||||
p = sub.add_parser("who", help="who has worked on a scope (notes, reviews, claims, messages)")
|
||||
p.add_argument("scope")
|
||||
p = sub.add_parser("re", help="reverse-engineering lookups from the project's image and annotations")
|
||||
p.add_argument("action", choices=["disasm", "xrefs", "bytes", "lookup", "ghidra", "ghidra-tools", "debugger-tools", "match", "similar", "struct-audit", "behaviour"])
|
||||
p.add_argument("action", choices=["disasm", "xrefs", "bytes", "lookup", "summary", "ghidra", "ghidra-tools", "ghidra-check", "debugger-tools", "match", "similar", "struct-audit", "behaviour"])
|
||||
p.add_argument("target", nargs="?", default="", help="an address (0x4AD850) or a name; for ghidra, the tool")
|
||||
p.add_argument("arguments", nargs="?", default="{}", help="ghidra: the tool's arguments as JSON")
|
||||
p.add_argument("--decomp", help="match: the checkout or worktree to compile from")
|
||||
@@ -722,6 +722,21 @@ def main(argv=None):
|
||||
elif args.action == "ghidra-tools":
|
||||
res = Ghidra(harbor.project.reverse).tools()
|
||||
out(res, "\n".join(f"{t['name']:<26} {t['description'].splitlines()[0][:90]}" for t in res))
|
||||
elif args.action == "ghidra-check":
|
||||
from .reverse import ghidra_check
|
||||
res = ghidra_check(harbor.project.reverse)
|
||||
mark = lambda ok: "ok " if ok else ("?? " if ok is None else "FAIL")
|
||||
out(res, "\n".join(
|
||||
[f"{'Ghidra serves our program' if res['ok'] else 'Ghidra does NOT serve our program'}: "
|
||||
f"{res['program']} ({res['functions_in_ghidra']} functions); image {res['image_base']}, "
|
||||
f"{res['image_size']} bytes, sha256 {res['image_sha256'][:16]}..."] +
|
||||
[f" {mark(c['ok'])} {c['check']}: " + (f"expected {c['expected']}, Ghidra {c['ghidra']}" if "ghidra" in c
|
||||
else ("differs at " + ", ".join(c["differ_at"]) if c.get("differ_at") else "all equal"))
|
||||
for c in res["checks"]]))
|
||||
elif args.action == "summary":
|
||||
from .reverse import format_summary
|
||||
res = r.summary(args.target)
|
||||
out(res, format_summary(res))
|
||||
elif args.action == "ghidra":
|
||||
text = Ghidra(harbor.project.reverse).call(args.target, json.loads(args.arguments))
|
||||
out({"result": text}, text)
|
||||
|
||||
+11
-1
@@ -709,7 +709,14 @@ class Harbor(WorkMixin, ExtrasMixin, FlowMixin, LaunchMixin):
|
||||
if kind == "draft" and lo is not None:
|
||||
bands = [a for a in active if a["kind"] == "band"
|
||||
and scopes_overlap((canonical, lo, hi), (a["scope"], a["lo"], a["hi"]))]
|
||||
if bands and not any(b["agent"] == agent for b in bands):
|
||||
# a task a coordinator assigned to this agent for this scope is the exception to another's band
|
||||
assigned = any(scopes_overlap((canonical, lo, hi), (t["scope"], *parse_scope(t["scope"])[1:]))
|
||||
for t in c.execute("SELECT scope FROM tasks WHERE assignee=? AND scope IS NOT NULL "
|
||||
"AND state IN ('open','in_progress') AND created_by IN (%s)"
|
||||
% ",".join("?" * len(self.coordinators)),
|
||||
(agent, *self.coordinators)).fetchall()
|
||||
if t["scope"].startswith(("fn:", "range:", "file:")))
|
||||
if bands and not any(b["agent"] == agent for b in bands) and not assigned:
|
||||
conflicts += bands
|
||||
if conflicts and not override:
|
||||
described = "; ".join(f"{a['agent']} holds {a['kind']} {a['scope']}"
|
||||
@@ -762,6 +769,7 @@ class Harbor(WorkMixin, ExtrasMixin, FlowMixin, LaunchMixin):
|
||||
raise HarborError("no reviewer: name one, or set reviewers in the project file")
|
||||
canonical = parse_scope(scope)[0] if scope else None
|
||||
commit = self.resolve_commit(author, commit)
|
||||
self.check_submit_policy(author)
|
||||
|
||||
def do(c):
|
||||
now = self.now()
|
||||
@@ -825,6 +833,8 @@ class Harbor(WorkMixin, ExtrasMixin, FlowMixin, LaunchMixin):
|
||||
self.check_landing_policy(item_id)
|
||||
if is_author and commit:
|
||||
commit = self.resolve_commit(agent, commit)
|
||||
if is_author and to_state == "submitted":
|
||||
self.check_submit_policy(agent)
|
||||
|
||||
def do(c):
|
||||
now = self.now()
|
||||
|
||||
@@ -265,6 +265,20 @@ class FlowMixin:
|
||||
raise HarborError(f"policy: item #{item_id} needs approval from {', '.join(missing)} before it lands "
|
||||
"(since its last submission)")
|
||||
|
||||
def check_submit_policy(self, author):
|
||||
"""policy submits_per_hour: an agent capped at N submissions (new or resubmitted) per hour must slow down
|
||||
and check its work before sending more."""
|
||||
cap = (self.policy("submits_per_hour") or {}).get(author)
|
||||
if not cap:
|
||||
return
|
||||
since = iso(utc(self.clock() - 3600))
|
||||
count = self.conn.execute("SELECT COUNT(*) FROM item_events WHERE agent=? AND to_state='submitted' AND at>=?",
|
||||
(author, since)).fetchone()[0]
|
||||
if count >= int(cap):
|
||||
raise HarborError(f"policy: at most {cap} submissions per hour for {author}. Slow down: re-read the "
|
||||
"disassembly against your C line by line, run quick_readable and re_behaviour, list "
|
||||
"your changed annotation rows, and submit only work you have checked twice")
|
||||
|
||||
def check_message_policy(self, sender, to_all):
|
||||
cap = self.policy("all_per_hour")
|
||||
if not cap or not to_all or sender in self.coordinators:
|
||||
|
||||
@@ -0,0 +1,69 @@
|
||||
"""The server's background compiles (the Logbook's statistics, the whole-program check, the emulator run) give way to
|
||||
the lander: its gates (a full readable check of every file) are what land work on main, and a second compile of main
|
||||
beside them roughly quadruples their time.
|
||||
|
||||
Two parts:
|
||||
lander_busy(project) while [logbook] lander_lock names a file that exists and was written in the last
|
||||
STALE_MINUTES, the lander is running its gates: background jobs do not start. The lander
|
||||
creates the file before its gates and removes it after the push; an older file is a lander
|
||||
that died, and is ignored.
|
||||
run_low(...) subprocess.run at BELOW_NORMAL priority (children such as cl.exe inherit it) that, on a
|
||||
timeout, ends the whole process tree instead of leaving the compiler running orphaned.
|
||||
|
||||
Configured per project in the [logbook] table:
|
||||
lander_lock = 'R:/Workspace/Data/lander-busy.lock' unset or "": never wait"""
|
||||
import os
|
||||
import subprocess
|
||||
import time
|
||||
from pathlib import Path
|
||||
|
||||
STALE_MINUTES = 60
|
||||
BELOW_NORMAL = getattr(subprocess, "BELOW_NORMAL_PRIORITY_CLASS", 0)
|
||||
IDLE = getattr(subprocess, "IDLE_PRIORITY_CLASS", 0)
|
||||
NO_WINDOW = getattr(subprocess, "CREATE_NO_WINDOW", 0)
|
||||
|
||||
|
||||
def lock_path(project):
|
||||
path = ((getattr(project, "logbook", None) or {}).get("lander_lock") or "").strip()
|
||||
return Path(path) if path else None
|
||||
|
||||
|
||||
def lander_busy(project, now=None):
|
||||
"""Why background work should wait (the lander's lock is fresh), or None."""
|
||||
path = lock_path(project)
|
||||
if path is None:
|
||||
return None
|
||||
try:
|
||||
written = path.stat().st_mtime
|
||||
except OSError:
|
||||
return None
|
||||
age = (time.time() if now is None else now) - written
|
||||
if age >= STALE_MINUTES * 60:
|
||||
return None
|
||||
return f"the lander is running its gates ({path.name}, {max(0, int(age // 60))} min old)"
|
||||
|
||||
|
||||
def kill_tree(pid):
|
||||
if os.name == "nt":
|
||||
subprocess.run(["taskkill", "/PID", str(pid), "/T", "/F"], capture_output=True, creationflags=NO_WINDOW)
|
||||
else:
|
||||
try:
|
||||
os.kill(pid, 9)
|
||||
except OSError:
|
||||
pass
|
||||
|
||||
|
||||
def run_low(argv, timeout, priority=BELOW_NORMAL, **kwargs):
|
||||
"""subprocess.run(argv, capture_output=True, text=True, ...) at a lower priority class; on a timeout the whole
|
||||
tree (cmd.exe, cl.exe) is ended before TimeoutExpired is raised."""
|
||||
flags = kwargs.pop("creationflags", 0) | (priority | NO_WINDOW if os.name == "nt" else 0)
|
||||
with subprocess.Popen(argv, stdin=subprocess.DEVNULL, stdout=subprocess.PIPE, stderr=subprocess.PIPE,
|
||||
text=True, creationflags=flags, **kwargs) as proc:
|
||||
try:
|
||||
stdout, stderr = proc.communicate(timeout=timeout)
|
||||
except subprocess.TimeoutExpired:
|
||||
kill_tree(proc.pid)
|
||||
proc.kill()
|
||||
proc.communicate()
|
||||
raise subprocess.TimeoutExpired(argv, timeout) from None
|
||||
return subprocess.CompletedProcess(argv, proc.returncode, stdout, stderr)
|
||||
+20
-7
@@ -9,6 +9,8 @@ Configured per project in a [logbook] table:
|
||||
decomp = 'R:/.../open-seas-decomp' the repository whose main branch is followed
|
||||
stats_repo = 'R:/.../stats-work/repo' a scratch copy of open-seas holding tools/progress/*.py
|
||||
interval_minutes = 10 how often the server looks (default 10)
|
||||
lander_lock = 'R:/.../lander-busy.lock' wait while the lander runs its gates (harbor.lander_gate)
|
||||
The progress tools run below normal priority.
|
||||
Milestones and narrative entries stay with people: they need judgement.
|
||||
|
||||
State (the last commit seen) is kept in the logbook folder as auto.json."""
|
||||
@@ -22,9 +24,10 @@ import time
|
||||
import traceback
|
||||
from pathlib import Path
|
||||
|
||||
from . import logbook
|
||||
from . import lander_gate, logbook
|
||||
|
||||
MAX_ENTRIES = 25
|
||||
STATS_TIMEOUT = 1800 # each progress tool; sync_decomp compiles every readable file, below normal priority
|
||||
|
||||
|
||||
def _git(repo, *args):
|
||||
@@ -73,8 +76,8 @@ def log_commits(project, repo, since, head):
|
||||
for sha, when, subject, body in commits:
|
||||
files = [f for f in _git(repo, "show", "--name-only", "--format=", sha).splitlines() if f.strip()]
|
||||
lines = [l for l in body.strip().splitlines() if l.strip() and not l.lower().startswith("co-authored-by")]
|
||||
detail = "\n".join(lines[:8]) if lines else subject
|
||||
detail += f"\n(main {sha[:8]}, {len(files)} file{'s' if len(files) != 1 else ''})"
|
||||
detail = "\n".join(lines[:8]) + "\n" if lines else "" # the board shows the title already
|
||||
detail += f"(main {sha[:8]}, {len(files)} file{'s' if len(files) != 1 else ''})"
|
||||
added.append(logbook.add_entry(project, subject, detail, kind=_kind(files), at=_utc(when)))
|
||||
return added
|
||||
|
||||
@@ -88,8 +91,10 @@ def rebuild_stats(project, repo, stats_repo, head):
|
||||
installer = r"C:\Program Files (x86)\Microsoft Visual Studio\Installer"
|
||||
if os.path.isdir(installer) and installer.lower() not in env.get("PATH", "").lower():
|
||||
env["PATH"] = installer + os.pathsep + env.get("PATH", "")
|
||||
run = lambda *argv: subprocess.run([sys.executable, "-B", *map(str, argv)], cwd=str(stats_repo), env=env,
|
||||
capture_output=True, text=True, encoding="utf-8", errors="replace", timeout=900)
|
||||
# below normal priority (cl.exe inherits it), so the lander's and the agents' compiles go first; a timeout ends
|
||||
# the whole tree, not only python (an orphaned cl.exe went on compiling all of main at normal priority)
|
||||
run = lambda *argv: lander_gate.run_low([sys.executable, "-B", *map(str, argv)], STATS_TIMEOUT,
|
||||
cwd=str(stats_repo), env=env, encoding="utf-8", errors="replace")
|
||||
sync = run(tools / "sync_decomp.py", "--decomp", repo)
|
||||
if sync.returncode != 0:
|
||||
raise RuntimeError("sync_decomp: " + (sync.stdout + sync.stderr).strip()[-400:])
|
||||
@@ -134,12 +139,20 @@ def refresh(project, force=False):
|
||||
state = _state(project)
|
||||
if state.get("last_head") == head and not force:
|
||||
return {"changed": False, "head": head[:8]}
|
||||
busy = None if force else lander_gate.lander_busy(project)
|
||||
if busy: # its gates compile all of main: look again next time
|
||||
return {"changed": False, "head": head[:8], "waiting": busy}
|
||||
out = {"changed": True, "head": head[:8]}
|
||||
if state.get("last_head"):
|
||||
# last_head is where the statistics are; logged_head where the ship's log is (a failed statistics run is retried,
|
||||
# and must not log the same commits twice)
|
||||
logged = state.get("logged_head") or state.get("last_head")
|
||||
if logged and logged != head:
|
||||
try:
|
||||
out["entries"] = len(log_commits(project, repo, state["last_head"], head))
|
||||
out["entries"] = len(log_commits(project, repo, logged, head))
|
||||
except RuntimeError as error: # e.g. the old commit vanished after a rewrite
|
||||
out["log_error"] = str(error)
|
||||
if logged:
|
||||
state["logged_head"] = head
|
||||
if stats_repo:
|
||||
try:
|
||||
out["stats"] = rebuild_stats(project, snapshot_at(config, repo, head), stats_repo, head)
|
||||
|
||||
@@ -38,6 +38,8 @@ review queue, presence and stop flags. Protocol:
|
||||
10. Before your session ends or your allowance runs out, write handoff(): finished, in flight, next.
|
||||
11. To read the original machine code use re_disassemble, re_xrefs, re_bytes and re_lookup: they read the project's
|
||||
own image with the team's current names, so do not write your own disassembler or offset maths.
|
||||
re_summary(address) gives the overview in one call: name, prototype, size, convention, readable status,
|
||||
callers and callees with their status, globals, strings, vtable slots and the Mac/Xbox hints.
|
||||
For Ghidra's view (decompiler, p-code, stack frames, struct field references, types) use ghidra_tools() once,
|
||||
then ghidra(tool, arguments): one shared read-only Ghidra session holding the team's annotations.
|
||||
To check a value in the running game (x87 control word, `this`, struct fields), use debugger_tools() and
|
||||
@@ -563,6 +565,14 @@ def re_lookup(target: str) -> dict:
|
||||
return _re(lambda r: r.lookup(target))
|
||||
|
||||
|
||||
@mcp.tool
|
||||
def re_summary(target: str) -> dict:
|
||||
"""One call before starting on a function: its current name and prototype, size and calling convention (read
|
||||
from the bytes), whether it is readable on main, its callers and callees with their names and whether each is
|
||||
readable on main, the globals and strings it uses, the vtable slots it fills, and the Mac/Xbox name hints."""
|
||||
return _re(lambda r: r.summary(target))
|
||||
|
||||
|
||||
_ghidra = {}
|
||||
|
||||
|
||||
@@ -692,6 +702,17 @@ def ghidra_tools() -> list:
|
||||
return _gh(lambda g: g.tools())
|
||||
|
||||
|
||||
@mcp.tool()
|
||||
def ghidra_check() -> dict:
|
||||
"""Is the shared Ghidra session serving our program? Compares its address range with the project image's base
|
||||
and size, the SHA-256 of the file it imported with ours, and sample bytes; ok is false when any differs."""
|
||||
try:
|
||||
from .reverse import ghidra_check as check
|
||||
return check(harbor().project.reverse)
|
||||
except (HarborError, projects.ProjectError) as error:
|
||||
raise ToolError(str(error)) from None
|
||||
|
||||
|
||||
@mcp.tool()
|
||||
def ghidra(tool: str, arguments: dict | None = None) -> str:
|
||||
"""Run one tool in the shared read-only Ghidra session, e.g. ghidra('get_code', {'function': 'GetFogLevel',
|
||||
|
||||
+11
-1
@@ -213,6 +213,7 @@ class ProgressMap:
|
||||
|
||||
units, unit_index, functions = [], {}, []
|
||||
game_functions = game_done = 0
|
||||
tiers = {t: {"functions": 0, "bytes": 0, "game_functions": 0, "game_bytes": 0} for t in STATUSES}
|
||||
for addr in sorted(facts):
|
||||
d = facts[addr]
|
||||
size = int(d.get("size") or 0)
|
||||
@@ -257,6 +258,11 @@ class ProgressMap:
|
||||
kind = "game" if counted else "library" if tier == "library" else "engine" if category == "engine" \
|
||||
else "unsorted" if category == "unsorted" else "library"
|
||||
u["kinds"][kind] = u["kinds"].get(kind, 0) + size
|
||||
tiers[tier]["functions"] += 1
|
||||
tiers[tier]["bytes"] += size
|
||||
if counted: # the Code bar's game functions, split by tier
|
||||
tiers[tier]["game_functions"] += 1
|
||||
tiers[tier]["game_bytes"] += size
|
||||
if counted:
|
||||
game_functions += 1
|
||||
game_done += done
|
||||
@@ -291,7 +297,11 @@ class ProgressMap:
|
||||
"source": "logbook symbol table" if symbols else "src headers and names.csv",
|
||||
"commit": commit, "main": main, "updated_at": stats.get("updated_at"),
|
||||
"percent": percent, "logbook_percent": stats.get("game_bytes_decompiled_percent"),
|
||||
"code": code, "data": self._data(facts, rev),
|
||||
"code": code, "data": self._data(facts, rev), "tiers": tiers,
|
||||
# every function in the executable (game, engine, platform, CRT), so engine work shows too
|
||||
"all_code": {"bytes": stats.get("bytes"), "functions": stats.get("functions"),
|
||||
"decompiled_bytes": ((stats.get("by_status") or {}).get("decompiled") or {}).get("bytes"),
|
||||
"decompiled_functions": ((stats.get("by_status") or {}).get("decompiled") or {}).get("functions")},
|
||||
"history": [[p.get("at"), p.get("decompiled"), p.get("commit"), p.get("game_bytes")]
|
||||
for p in history if p.get("decompiled") is not None],
|
||||
"statuses": list(STATUSES),
|
||||
|
||||
@@ -196,6 +196,14 @@ def _policy(raw, names):
|
||||
policy["all_per_hour"] = int(policy["all_per_hour"])
|
||||
except (TypeError, ValueError):
|
||||
raise ProjectError("policy all_per_hour must be a whole number")
|
||||
if "submits_per_hour" in policy: # {agent: n}: submissions and resubmissions per hour
|
||||
caps = policy["submits_per_hour"]
|
||||
if not isinstance(caps, dict) or not all(k in names for k in caps):
|
||||
raise ProjectError("policy submits_per_hour must map agents on the roster to whole numbers")
|
||||
try:
|
||||
policy["submits_per_hour"] = {k: int(v) for k, v in caps.items()}
|
||||
except (TypeError, ValueError):
|
||||
raise ProjectError("policy submits_per_hour must map agents on the roster to whole numbers")
|
||||
return policy
|
||||
|
||||
|
||||
|
||||
@@ -207,6 +207,10 @@ class FunctionQueue:
|
||||
head = self.r.read(addr, 10) if int(facts[addr].get("size") or 0) == 10 else b""
|
||||
if head[:1] and 0xB8 <= head[0] <= 0xBF and head[5:6] == b"\xe9":
|
||||
continue # mov reg, imm32; jmp: an at-exit destructor thunk or an SEH handler stub (compiler's)
|
||||
first = self.r.read(addr, 3)
|
||||
if len(first) == 3 and first[0] in (0x8B, 0x8D, 0xFF, 0x83, 0xC7, 0x89) and (first[1] & 0xC7) == 0x45 \
|
||||
and first[2] >= 0x80:
|
||||
continue # opens on the parent's [ebp-x] with no frame: a catch or unwind funclet (ruling #12777)
|
||||
score, reasons = self.score(addr, facts, named, rejected, similar_readable, with_convention=False)
|
||||
ranked.append((score, addr, reasons))
|
||||
ranked.sort(reverse=True)
|
||||
|
||||
+14
-12
@@ -13,6 +13,7 @@ Configured per project in the [logbook] table (main is [logbook] decomp):
|
||||
behaviour_every_hours = 12
|
||||
behaviour_window = "01-06" from 01:00 to 06:00 local time; "" for any hour
|
||||
|
||||
No run starts while the lander runs its gates ([logbook] lander_lock, harbor.lander_gate).
|
||||
One run at a time (the server runs one thread for every project), at the lowest process priority, into a .partial file
|
||||
that replaces the results only once it is complete. The match run holds the same per-checkout lock as re_match_program,
|
||||
so it waits its turn behind any other whole-program build of main.
|
||||
@@ -31,7 +32,7 @@ import time
|
||||
import traceback
|
||||
from pathlib import Path
|
||||
|
||||
from . import logbook
|
||||
from . import lander_gate, logbook
|
||||
|
||||
JOBS = ("match", "behaviour")
|
||||
RESULTS_KEY = {"match": "match_results", "behaviour": "diffrun_results"}
|
||||
@@ -180,14 +181,7 @@ def _spawn(argv, cwd, stdout, stderr):
|
||||
|
||||
|
||||
def _kill_tree(pid):
|
||||
if os.name == "nt":
|
||||
subprocess.run(["taskkill", "/PID", str(pid), "/T", "/F"], capture_output=True,
|
||||
creationflags=getattr(subprocess, "CREATE_NO_WINDOW", 0))
|
||||
else:
|
||||
try:
|
||||
os.kill(pid, 9)
|
||||
except OSError:
|
||||
pass
|
||||
lander_gate.kill_tree(pid)
|
||||
|
||||
|
||||
def _alive(pid):
|
||||
@@ -205,12 +199,12 @@ def _paths(project, job):
|
||||
return paths
|
||||
|
||||
|
||||
def _argv(project, job, out):
|
||||
def _argv(project, job, out, decomp=None):
|
||||
need = configured(project, job)
|
||||
if job == "match":
|
||||
script = Path(need["script"])
|
||||
return [sys.executable, "-B", str(script), "check", "--build", "--decomp",
|
||||
str(Path(project.logbook["decomp"])), "--json", str(out)], script.parent
|
||||
str(Path(decomp or project.logbook["decomp"])), "--json", str(out)], script.parent
|
||||
command = [str(c) for c in need["command"]]
|
||||
return command + ["--readable", "--jobs", "4", "--json", str(out)], Path(command[-1]).parent
|
||||
|
||||
@@ -260,7 +254,12 @@ def _complete(project, state, run, now, returncode=None):
|
||||
|
||||
def run_job(project, state, job, head, now, reason=""):
|
||||
paths = _paths(project, job)
|
||||
argv, cwd = _argv(project, job, paths["partial"])
|
||||
decomp = None
|
||||
if job == "match" and project.logbook.get("results_snapshot") and head:
|
||||
# a clean checkout of exactly `head`: the live checkout is the lander's and changes under a running build
|
||||
from .logbook_auto import snapshot_at
|
||||
decomp = snapshot_at({"snapshot": project.logbook["results_snapshot"]}, project.logbook["decomp"], head)
|
||||
argv, cwd = _argv(project, job, paths["partial"], decomp)
|
||||
for stale in [paths["partial"]] + ([paths["text_partial"]] if job == "behaviour" else []):
|
||||
if stale.exists():
|
||||
stale.write_bytes(b"") # a leftover of an interrupted run is never taken for this one
|
||||
@@ -310,6 +309,9 @@ def tick(project, now=None, hour=None):
|
||||
job, why = choose(settings(project), state, head, now, hour, requested, available)
|
||||
if not job:
|
||||
return {"idle": why, "head": head[:8]}
|
||||
busy = lander_gate.lander_busy(project, now)
|
||||
if busy: # a request is kept for the next look
|
||||
return {"waiting": busy, "job": job}
|
||||
if job == "match":
|
||||
from .util import HarborError
|
||||
try:
|
||||
|
||||
@@ -7,6 +7,8 @@ Configured per project in a [reverse] table:
|
||||
imports = 'R:/.../imports.json' optional: [{module, name, slot}]
|
||||
annotations = 'R:/.../annotations' names.csv, functions.csv, globals.csv, prototypes.h
|
||||
src = 'R:/.../src' optional: the decompiled C files (<ADDR>_<Name>.c)
|
||||
sections = 'R:/.../sections.json' optional: [{name, start, end}] (default: sections.json beside the image),
|
||||
used to find the vtables and pointer tables a function sits in
|
||||
|
||||
Every answer comes from those files, read fresh when they change, so the agents see the team's current names.
|
||||
Nothing here writes anything."""
|
||||
@@ -42,6 +44,7 @@ class Reverse:
|
||||
self.conventions_path = Path(config["conventions"]) if config.get("conventions") else None
|
||||
self.source_files = Path(config["source_files"]) if config.get("source_files") else None
|
||||
self.mac_names = Path(config["mac_names"]) if config.get("mac_names") else None
|
||||
self.sections_path = Path(config["sections"]) if config.get("sections") else None
|
||||
self._cache = {}
|
||||
|
||||
# ----------------------------------------------------------------- loading (cached by file mtime)
|
||||
@@ -491,6 +494,176 @@ class Reverse:
|
||||
break
|
||||
return out
|
||||
|
||||
# ----------------------------------------------------------------- one-call summary
|
||||
def _src_index(self):
|
||||
"""address -> path of src/**/<ADDR>_<Name>.c (one directory walk, reused for 30 seconds)."""
|
||||
import time
|
||||
hit = self._cache.get("src_index")
|
||||
if hit and time.monotonic() - hit[0] < 30:
|
||||
return hit[1]
|
||||
out = {}
|
||||
if self.src and self.src.exists():
|
||||
for folder, _, files in os.walk(self.src):
|
||||
for name in files:
|
||||
m = re.match(r"([0-9A-Fa-f]{6,8})_.*\.c$", name)
|
||||
if m:
|
||||
out.setdefault(int(m.group(1), 16), Path(folder) / name)
|
||||
self._cache["src_index"] = (time.monotonic(), out)
|
||||
return out
|
||||
|
||||
def status_of(self, addr):
|
||||
"""The decompiled file's status on main ('readable', 'raw', ...), or None when there is no file."""
|
||||
path = self._src_index().get(addr)
|
||||
if not path:
|
||||
return None
|
||||
try:
|
||||
with path.open(encoding="utf-8", errors="replace") as f:
|
||||
head = [next(f, "") for _ in range(8)]
|
||||
except OSError:
|
||||
return None
|
||||
return next((l.split("status:", 1)[1].strip() for l in head if "status:" in l), "raw")
|
||||
|
||||
def _sections(self):
|
||||
"""[(name, start, end)] from sections.json beside the image (or [reverse] sections), else []."""
|
||||
path = self.sections_path or self.image.with_name("sections.json")
|
||||
return self._load("sections", path, lambda p: [
|
||||
(s["name"], _int(s["start"]), _int(s["end"])) for s in json.loads(p.read_text(encoding="utf-8"))] if p else [])
|
||||
|
||||
def _code_data_refs(self):
|
||||
facts = self._facts()
|
||||
key = ("data_refs", id(facts))
|
||||
if key not in self._cache:
|
||||
self._cache[key] = {_int(a) for d in facts.values() for a in d.get("data", [])}
|
||||
return self._cache[key]
|
||||
|
||||
def _mac_vtables(self):
|
||||
return self._load("mac_vtables", self.mac_names.with_name("vtables.csv") if self.mac_names else None,
|
||||
lambda p: {int(r["pc_vtable"], 16): r.get("class") for r in
|
||||
csv.DictReader(p.read_text(encoding="utf-8").splitlines())
|
||||
if (r.get("pc_vtable") or "").startswith("0x")} if p else {})
|
||||
|
||||
def vtable_slots(self, addr):
|
||||
"""Where the function's address sits in a table of code pointers outside .text: the table start (walked back
|
||||
over function addresses to the entry the code references), the slot index, and what the table is."""
|
||||
data = self._bytes()
|
||||
facts = self._facts()
|
||||
sections = self._sections()
|
||||
text = next(((lo, hi) for name, lo, hi in sections if name == ".text"), None)
|
||||
tables = [(lo, hi) for name, lo, hi in sections if name in (".rdata", ".data")]
|
||||
if not tables: # no section list: everything past the last function
|
||||
end = max((a + int(d.get("size") or 0) for a, d in facts.items()), default=self.base)
|
||||
tables = [(end, self.base + len(data))]
|
||||
refs, mac = self._code_data_refs(), self._mac_vtables()
|
||||
is_code = (lambda v: v in facts) if facts else (lambda v: text and text[0] <= v < text[1])
|
||||
needle = struct.pack("<I", addr)
|
||||
out = []
|
||||
for lo, hi in tables:
|
||||
pos = data.find(needle, max(0, lo - self.base), hi - self.base)
|
||||
while pos >= 0 and len(out) < 40:
|
||||
if pos % 4 == 0:
|
||||
start = pos
|
||||
while start - 4 >= lo - self.base and (start + self.base) not in refs and \
|
||||
(start + self.base) not in mac and \
|
||||
is_code(struct.unpack_from("<I", data, start - 4)[0]) and pos - start < 4 * 400:
|
||||
start -= 4
|
||||
table = start + self.base
|
||||
slot = (pos - start) // 4
|
||||
name = self.name_of(table)
|
||||
kind = "vtable" if table in refs or table in mac or (name and "vtable" in name.lower()) \
|
||||
else "pointer table"
|
||||
out.append({"table": f"0x{table:X}", "slot": slot, "offset": f"0x{slot * 4:X}", "kind": kind,
|
||||
**({"table_name": name} if name else {}),
|
||||
**({"mac_class": mac[table]} if table in mac else {})})
|
||||
pos = data.find(needle, pos + 1, hi - self.base)
|
||||
return out
|
||||
|
||||
def summary(self, target):
|
||||
"""Everything a worker needs before starting on a function, in one call: name, prototype, size, convention,
|
||||
whether it is readable on main, callers and callees (named, with their status on main), globals and strings,
|
||||
the vtable slots it fills, and the Mac/Xbox name hints."""
|
||||
addr = self.resolve(target)
|
||||
func = self.function_at(addr)
|
||||
if not func:
|
||||
raise HarborError(f"0x{addr:X} is not inside a known function")
|
||||
start, size, name = func
|
||||
d = self._facts()[start]
|
||||
|
||||
def fn(a):
|
||||
status = self.status_of(a)
|
||||
return {"address": f"0x{a:X}", "name": self.name_of(a), "status": status or "no file",
|
||||
"readable": status == "readable"}
|
||||
|
||||
out = {"address": f"0x{start:X}", "name": name, "size": size, "end": f"0x{start + size:X}"}
|
||||
if addr != start:
|
||||
out["asked"] = f"0x{addr:X} (+0x{addr - start:X})"
|
||||
proto = self._prototypes().get(name) if name else None
|
||||
out["prototype"] = proto
|
||||
conv = self.convention(start)
|
||||
out["convention"] = conv["summary"] if conv else None
|
||||
if conv:
|
||||
out["registers_in"], out["ret_pops"] = conv["registers_in"], conv["ret_pops"]
|
||||
status = self.status_of(start)
|
||||
out["status"] = status or "no file"
|
||||
out["readable"] = status == "readable"
|
||||
path = self._src_index().get(start)
|
||||
if path and self.src:
|
||||
out["file"] = str(path.relative_to(self.src.parent)).replace(os.sep, "/")
|
||||
row = self._csv("names.csv").get(start)
|
||||
if row and len(row) > 3 and row[3]:
|
||||
out["note"] = row[3] if len(row[3]) < 600 else row[3][:600] + "..."
|
||||
callers = sorted(set(self._callers().get(start, [])))
|
||||
callees = list(dict.fromkeys(_int(a) for a in d.get("calls", [])))
|
||||
out["callers"] = [fn(a) for a in callers]
|
||||
out["callees"] = [fn(a) for a in callees]
|
||||
if d.get("imports"):
|
||||
out["imports"] = d["imports"]
|
||||
glob = self._csv("globals.csv")
|
||||
out["globals"] = [{"address": f"0x{a:X}", "name": self.name_of(a),
|
||||
**({"type": glob[a][2]} if a in glob and len(glob[a]) > 2 and glob[a][2] else {})}
|
||||
for a in dict.fromkeys(_int(x) for x in d.get("data", []))]
|
||||
out["strings"] = d.get("strings", [])
|
||||
out["vtable_slots"] = self.vtable_slots(start)
|
||||
out["hints"] = self.origins(start)
|
||||
out["counts"] = {"callers": len(callers), "callers_readable": sum(c["readable"] for c in out["callers"]),
|
||||
"callees": len(callees), "callees_readable": sum(c["readable"] for c in out["callees"])}
|
||||
return out
|
||||
|
||||
|
||||
def format_summary(s):
|
||||
"""Reverse.summary() as text for the command line."""
|
||||
mark = lambda f: f"{f['address']} {f['name'] or '(unnamed)'} [{'readable' if f['readable'] else f['status']}]"
|
||||
lines = [f"{s['name'] or '(unnamed)'} {s['address']}..{s['end']} ({s['size']} bytes)"
|
||||
+ (f", asked {s['asked']}" if s.get("asked") else ""),
|
||||
f"status on main: {s['status']}" + (f" ({s['file']})" if s.get("file") else ""),
|
||||
f"convention: {s['convention'] or 'unknown'}",
|
||||
"prototype: " + (s["prototype"] or "none in prototypes.h")]
|
||||
c = s["counts"]
|
||||
lines.append(f"callers ({c['callers']}, {c['callers_readable']} readable):")
|
||||
lines += [f" {mark(f)}" for f in s["callers"]] or [" none"]
|
||||
lines.append(f"callees ({c['callees']}, {c['callees_readable']} readable):")
|
||||
lines += [f" {mark(f)}" for f in s["callees"]] or [" none"]
|
||||
if s.get("imports"):
|
||||
lines.append("imports: " + ", ".join(s["imports"]))
|
||||
lines.append(f"globals ({len(s['globals'])}):")
|
||||
lines += [f" {g['address']} {g['name'] or '(unnamed)'}" + (f" : {g['type']}" if g.get("type") else "")
|
||||
for g in s["globals"]] or [" none"]
|
||||
if s["strings"]:
|
||||
lines.append("strings: " + ", ".join(repr(x) for x in s["strings"]))
|
||||
lines.append("vtable slots:" + ("" if s["vtable_slots"] else " none"))
|
||||
lines += [f" {v['kind']} {v['table']}" + (f" {v['table_name']}" if v.get("table_name") else "")
|
||||
+ (f" (Mac class {v['mac_class']})" if v.get("mac_class") else "") + f" slot {v['slot']} (+{v['offset']})"
|
||||
for v in s["vtable_slots"]]
|
||||
h = s["hints"]
|
||||
if h.get("mac_name"):
|
||||
lines.append(f"Mac name: {h['mac_name'].get('demangled')} ({h['mac_name'].get('confidence')})")
|
||||
if h.get("source_file"):
|
||||
sf = h["source_file"]
|
||||
lines.append(f"Xbox source file: {sf.get('source_file')}" + (f" lines {sf['lines']}" if sf.get("lines") else "")
|
||||
+ f" ({sf.get('confidence')})")
|
||||
if not h:
|
||||
lines.append("Mac/Xbox hints: none")
|
||||
return "\n".join(lines)
|
||||
|
||||
|
||||
class Ghidra:
|
||||
"""A small client for the shared, read-only Ghidra MCP server (GhidrAssistMCP over streamable HTTP), so every
|
||||
@@ -572,6 +745,63 @@ class Ghidra:
|
||||
return re.sub(r"^\[Context\] Operating on: [^\n]*\n+", "", text)
|
||||
|
||||
|
||||
def _hexdump_bytes(text):
|
||||
"""The bytes of a GhidrAssistMCP get_data_at hexdump ('401000 53 55 8b ... |SU.|')."""
|
||||
out = bytearray()
|
||||
for line in text.splitlines():
|
||||
m = re.match(r"^\s*[0-9A-Fa-f]{4,16}\s{2}((?:[0-9A-Fa-f]{2}\s{1,2})+)", line + " ")
|
||||
if m:
|
||||
out += bytes.fromhex("".join(m.group(1).split()))
|
||||
return bytes(out)
|
||||
|
||||
|
||||
def ghidra_check(config, samples=16, sample_len=128):
|
||||
"""Is the shared Ghidra server serving our program? Read-only: compares the program's address range with the
|
||||
project image (base and size), the SHA-256 of the file Ghidra imported (when that path is readable here) with
|
||||
the image's, and sample bytes read through Ghidra (get_data_at) with the image at spread-out addresses."""
|
||||
import hashlib
|
||||
reverse = Reverse(config)
|
||||
image = reverse._bytes()
|
||||
if not image:
|
||||
raise HarborError(f"the project image {reverse.image} is missing")
|
||||
base = reverse.base
|
||||
g = Ghidra(config)
|
||||
info = g.call("get_binary_info", {})
|
||||
binaries = g.call("list_binaries", {})
|
||||
field = lambda label, text: (re.search(rf"^\s*{label}:\s*(.+?)\s*$", text, re.M) or [None, None])[1]
|
||||
lo, hi = field("Min Address", info), field("Max Address", info)
|
||||
checks = []
|
||||
want_lo, want_hi = base, base + len(image) - 1
|
||||
got = (int(lo, 16), int(hi, 16)) if lo and hi else None
|
||||
checks.append({"check": "address range", "ok": got == (want_lo, want_hi),
|
||||
"expected": f"0x{want_lo:X}..0x{want_hi:X} ({len(image)} bytes)",
|
||||
"ghidra": f"0x{got[0]:X}..0x{got[1]:X} ({got[1] - got[0] + 1} bytes)" if got else info.strip()[:200]})
|
||||
sha = hashlib.sha256(image).hexdigest()
|
||||
exe = field("Executable Path", binaries)
|
||||
local = Path(exe[1:] if exe and re.match(r"^/[A-Za-z]:/", exe) else exe) if exe else None
|
||||
if local and local.is_file():
|
||||
theirs = hashlib.sha256(local.read_bytes()).hexdigest()
|
||||
checks.append({"check": "sha256 of the imported file", "ok": theirs == sha, "expected": sha,
|
||||
"ghidra": theirs, "file": str(local)})
|
||||
else:
|
||||
checks.append({"check": "sha256 of the imported file", "ok": None, "expected": sha,
|
||||
"ghidra": f"not readable here ({exe or 'no path reported'})"})
|
||||
step = max(sample_len, (len(image) - sample_len) // max(1, samples - 1)) & ~0xF
|
||||
bad = []
|
||||
for i in range(samples):
|
||||
addr = base + min(i * step, len(image) - sample_len)
|
||||
theirs = _hexdump_bytes(g.call("get_data_at", {"address": f"0x{addr:X}", "len": sample_len}))
|
||||
ours = image[addr - base:addr - base + sample_len]
|
||||
if theirs != ours:
|
||||
bad.append(f"0x{addr:X}")
|
||||
checks.append({"check": f"{samples} byte samples of {sample_len}", "ok": not bad,
|
||||
**({"differ_at": bad} if bad else {})})
|
||||
ok = all(c["ok"] is not False for c in checks)
|
||||
return {"ok": ok, "program": field("Name", info), "image": str(reverse.image), "image_sha256": sha,
|
||||
"image_base": f"0x{base:X}", "image_size": len(image), "functions_in_ghidra": field("Function Count", info),
|
||||
"checks": checks}
|
||||
|
||||
|
||||
class Debugger:
|
||||
"""A per-agent x64dbg session through the x64dbg-automate MCP server (stdio), started on first use and kept for
|
||||
the life of this Harbor process, so breakpoints and the paused game survive between calls. Configured by
|
||||
|
||||
+32
-24
@@ -699,7 +699,7 @@ button.chip:hover { color: var(--accent-ink); background: var(--accent-soft); }
|
||||
<div class="lb-stamp" id="lbStamp">Loading…</div></div></section>
|
||||
<section class="card" aria-labelledby="h-pm" id="pmCard">
|
||||
<div class="body">
|
||||
<h2 class="pm-title" id="h-pm"><span id="pmProject">The project</span> is <b id="pmPct">–</b> decompiled</h2>
|
||||
<h2 class="pm-title" id="h-pm"><span id="pmProject">The project</span>: game code <b id="pmPct">–</b> decompiled</h2>
|
||||
<div class="pm-bars" id="pmBars"></div>
|
||||
<div class="note pm-fresh" id="pmFresh"></div>
|
||||
<div class="pm-commit"><span id="pmCommit"></span><span class="grow"></span>
|
||||
@@ -849,7 +849,7 @@ const ic=(n,c)=>'<svg class="i'+(c?' '+c:'')+'" aria-hidden="true"><use href="st
|
||||
const empty=(icon,title,text)=>'<div class="empty">'+ic(icon)+'<strong>'+esc(title)+'</strong><span>'+(text||'')+'</span></div>';
|
||||
const INIT={Claude:'CL',Codex:'CX',DeepSeek:'DS',OpenCode:'OC',Kimi:'KM',User:'You'};
|
||||
const ago=m=>m==null?'never':m<1?'just now':m<120?m+' min ago':m<2880?Math.round(m/60)+' h ago':Math.round(m/1440)+' days ago';
|
||||
const central=s=>{const d=new Date(s);return isNaN(d)?s:d.toLocaleString('en-US',{timeZone:'America/Chicago',month:'short',day:'numeric',hour:'numeric',minute:'2-digit',timeZoneName:'short'})};
|
||||
const central=s=>{const d=new Date(s);return isNaN(d)?s:d.toLocaleString('en-US',{timeZone:'America/Chicago',month:'short',day:'numeric',hour:'numeric',minute:'2-digit',hour12:true})+' CST'};
|
||||
const STATE={submitted:'blue',in_review:'violet',changes_requested:'amber',approved:'green',held:'',landed:'green',withdrawn:'',rejected:'red'};
|
||||
let data=null,openThread=null,current=null,doctorRan=false,unseen=0;
|
||||
const HUMAN=()=>data&&data.project?data.project.human:'User';
|
||||
@@ -1132,17 +1132,21 @@ el('menuBtn').onclick=()=>el('shell').classList.add('open');
|
||||
el('scrim').onclick=()=>el('shell').classList.remove('open');
|
||||
window.addEventListener('hashchange',()=>{const v=location.hash.slice(1);if(v&&v!==current)show(v)});
|
||||
const TIERS=[['exact','Exact match','byte-identical with the game’s own compiler'],['verified','Verified','behaves the same as the game in the emulator'],
|
||||
['readable','Readable','readable C that passed review'],['deferred','Deferred','set aside with a reason after failed attempts'],['raw','Raw','not decompiled yet'],
|
||||
['library','Library','C runtime, D3DX9 and other linked libraries: never decompiled'],['engine','Engine','Gamebryo engine code: names only']];
|
||||
['readable','Readable','readable C that passed review'],['deferred','Deferred','set aside with a reason after failed attempts'],['raw','Raw','game code not decompiled yet'],['unsorted','Unsorted','not classified yet, outside the known library ranges (the few inside them count as Library)'],
|
||||
['library','Library','C runtime, D3DX9 and other linked libraries: never decompiled'],['engine','Engine','Gamebryo engine code not decompiled: names only']];
|
||||
const fmt=n=>Number(n||0).toLocaleString('en-US');
|
||||
function renderDecomp(){renderProgress();renderNextUp();renderDeferred()}
|
||||
async function renderProgress(){let p;try{p=await get('api/decomp/progress')}catch(e){put('progress','<div class="empty err">'+esc(e.message)+'</div>');return}
|
||||
const t=p.tiers,own=['exact','verified','readable','deferred','raw'].reduce((a,k)=>a+t[k].functions,0),done=t.exact.functions+t.verified.functions+t.readable.functions;
|
||||
el('progressSum').textContent=fmt(done)+' of '+fmt(own)+' game functions decompiled';
|
||||
async function renderProgress(){let p;try{await pmLoad();p=PM.doc&&PM.doc.tiers?PM.doc:await get('api/decomp/progress')}catch(e){put('progress','<div class="empty err">'+esc(e.message)+'</div>');return}
|
||||
const t=tz(p.tiers),g=k=>t[k].game_functions!=null?t[k].game_functions:t[k].functions,
|
||||
own=['exact','verified','readable','deferred','raw'].reduce((a,k)=>a+g(k),0),done=g('exact')+g('verified')+g('readable');
|
||||
el('progressSum').textContent=fmt(done)+' of '+fmt(own)+' game functions decompiled · the bar shows every function';
|
||||
put('progress',tierBar(t)+'<div class="lb-legend">'+TIERS.map(([k,l])=>'<span><i class="tier-key tier-'+k+'"></i>'+esc(l)+' '+fmt(t[k].functions)+'</span>').join('')+'</div><div class="note" style="margin-top:10px">Bytes, categories, the trend and the ship’s log: <a href="#logbook" data-go="logbook">open the Logbook</a>.</div>');}
|
||||
function tierTable(t){return '<table style="margin-top:14px"><thead><tr><th>Tier</th><th style="text-align:right">Functions</th><th style="text-align:right">Bytes</th><th>Meaning</th></tr></thead><tbody>'
|
||||
+TIERS.map(([k,l,d])=>'<tr><td><span class="tier-key tier-'+k+'"></span>'+esc(l)+'</td><td class="num">'+fmt(t[k].functions)+'</td><td class="num">'+fmt(t[k].bytes)+'</td><td class="note">'+esc(d)+'</td></tr>').join('')+'</tbody></table>'}
|
||||
function tierBar(t){return '<div class="tierbar" role="img" aria-label="'+esc(TIERS.map(([k,l])=>l+' '+t[k].functions).join(', '))+'">'+TIERS.map(([k,l])=>t[k].functions?'<span class="tier-'+k+'" style="flex:'+t[k].functions+'" title="'+esc(l)+': '+fmt(t[k].functions)+' functions"></span>':'').join('')+'</div>'}
|
||||
const tz=t=>Object.fromEntries(TIERS.map(([k])=>[k,t[k]||{functions:0,bytes:0}]));
|
||||
function tierTable(t){t=tz(t);const g=k=>t[k].game_functions,has=g('raw')!=null,sum=f=>TIERS.reduce((a,[k])=>a+(f(k)||0),0);
|
||||
return '<table style="margin-top:14px"><thead><tr><th>Tier</th>'+(has?'<th style="text-align:right">Game functions</th>':'')+'<th style="text-align:right">All functions</th><th style="text-align:right">Bytes (all)</th><th>Meaning</th></tr></thead><tbody>'
|
||||
+TIERS.map(([k,l,d])=>'<tr><td><span class="tier-key tier-'+k+'"></span>'+esc(l)+'</td>'+(has?'<td class="num">'+fmt(g(k))+'</td>':'')+'<td class="num">'+fmt(t[k].functions)+'</td><td class="num">'+fmt(t[k].bytes)+'</td><td class="note">'+esc(d)+'</td></tr>').join('')
|
||||
+'<tr><td><b>Total</b></td>'+(has?'<td class="num"><b>'+fmt(sum(g))+'</b></td>':'')+'<td class="num"><b>'+fmt(sum(k=>t[k].functions))+'</b></td><td class="num"><b>'+fmt(sum(k=>t[k].bytes))+'</b></td><td class="note">exact + verified + readable = decompiled</td></tr></tbody></table>'}
|
||||
function tierBar(t){t=tz(t);return '<div class="tierbar" role="img" aria-label="'+esc(TIERS.map(([k,l])=>l+' '+t[k].functions).join(', '))+'">'+TIERS.map(([k,l])=>t[k].functions?'<span class="tier-'+k+'" style="flex:'+t[k].functions+'" title="'+esc(l)+': '+fmt(t[k].functions)+' functions"></span>':'').join('')+'</div>'}
|
||||
async function renderNextUp(){if(!el('nextUp')._html)put('nextUp',empty('clock','Ranking the bands…','This takes a few seconds the first time.'));let q;
|
||||
try{q=await get('api/decomp/queues')}catch(e){put('nextUp','<div class="empty err">'+esc(e.message)+'</div>');return}
|
||||
put('nextUp',q.length?q.map(a=>'<div class="band"><div class="cell-who">'+avatar(a.agent,'xs')+'<b>'+esc(a.agent)+'</b><span class="note mono">'+esc(a.bands.map(b=>b.join('–')).join(', '))+'</span></div>'
|
||||
@@ -1178,24 +1182,27 @@ const LB_CATS=[['game','Game code','Firaxis game logic: decompiled to readable C
|
||||
['platform','Platform','Miles, Bink, D3D9 and input wrappers: replaced in the port'],['crt','C runtime','MSVC 7.1 runtime: replaced by Rust’s standard library'],['unsorted','Unsorted','not classified yet; each lands in one of the four above']];
|
||||
let LB=null;
|
||||
const pct=n=>Number(n||0).toFixed(Number(n)>=10?1:2)+'%';
|
||||
const kb=b=>b>=1048576?(b/1048576).toFixed(2)+' MB':Math.round((b||0)/1024)+' KB';
|
||||
const kb=b=>b>=1e6?(b/1e6).toFixed(2)+' MB':Math.round((b||0)/1e3)+' kB';
|
||||
const cssVar=n=>getComputedStyle(el('view-logbook')).getPropertyValue(n).trim();
|
||||
function renderLbFigs(){const s=(LB&&LB.stats)||{},cat=k=>(s.by_category||{})[k]||{functions:0,bytes:0};
|
||||
/* by_status.decompiled counts every category (engine and platform too); the game-only count comes from the progress map */
|
||||
const total=s.functions||0,game=cat('game').functions,uns=cat('unsorted').functions,all=(s.by_status&&s.by_status.decompiled||{}).functions||0;
|
||||
const c=PM.doc&&PM.doc.code,done=c?c.decompiled_functions:null,handled=done==null?null:total-uns-Math.max(0,game-done);
|
||||
put('lbFigs',[[handled==null?'…':pct(total?100*handled/total:0)+' done',handled==null?'counting…':fmt(handled)+' of '+fmt(total)+' functions handled','game code decompiled, everything else identified'],
|
||||
[done==null?'…':fmt(done)+' of '+fmt(game),'game functions decompiled to readable C',done==null?fmt(all)+' decompiled across all code':fmt(game-done)+' to go · '+fmt(all)+' decompiled across all code'],
|
||||
[fmt(uns),'functions not sorted yet','each one is either game code (to decompile) or engine/library code (to name)']]
|
||||
.map(([v,l,n])=>'<div class="lb-fig"><div class="v">'+esc(v)+'</div><div class="l">'+esc(l)+'</div><div class="n">'+esc(n)+'</div></div>').join(''))}
|
||||
async function renderLogbook(){try{LB=await get('api/logbook')}catch(e){put('lbStamp','<span class="err">'+esc(e.message)+'</span>');return}
|
||||
const s=LB.stats||{},cat=k=>(s.by_category||{})[k]||{functions:0,bytes:0};
|
||||
const cur=LB.milestones.find(m=>/progress/i.test(m.state||''));el('lbMilestone').textContent=cur?'Milestone '+cur.code+' · '+cur.title:'Milestones';
|
||||
const cur=LB.milestones.filter(m=>/progress/i.test(m.state||'')).pop();el('lbMilestone').textContent=cur?'Milestone '+cur.code+' · '+cur.title:'Milestones';
|
||||
const au=LB.auto||{};put('lbStamp',(s.updated_at?'Statistics updated <b>'+esc(central(s.updated_at))+'</b><br>main at <b>'+esc(s.commit||'–')+'</b>':'No statistics yet')+(au.last_refresh?'<br>Keeps itself current: last checked '+esc(central(au.last_refresh))+(au.last_error?' · <span class="err">'+esc(String(au.last_error).slice(0,120))+'</span>':''):''));
|
||||
const total=s.functions||0,game=cat('game').functions,done=(s.by_status&&s.by_status.decompiled||{}).functions||0,uns=cat('unsorted').functions;
|
||||
const handled=total-uns-Math.max(0,game-done);
|
||||
put('lbFigs',[[pct(total?100*handled/total:0)+' done',fmt(handled)+' of '+fmt(total)+' functions handled','game code decompiled, everything else identified'],
|
||||
[fmt(done)+' of '+fmt(game),'game functions decompiled to readable C',fmt(game-done)+' to go'],
|
||||
[fmt(uns),'functions not sorted yet','each one is either game code (to decompile) or engine/library code (to name)']]
|
||||
.map(([v,l,n])=>'<div class="lb-fig"><div class="v">'+esc(v)+'</div><div class="l">'+esc(l)+'</div><div class="n">'+esc(n)+'</div></div>').join(''));
|
||||
renderLbFigs();
|
||||
put('lbLegend',LB_CATS.map(([k,n])=>'<span><i class="lb-sw'+(k==='unsorted'?' lb-hatch':'')+'" style="background:var(--lb-'+k+')"></i>'+esc(n)+'</span>').join('')+'<span><i class="lb-sw" style="background:var(--lb-done)"></i>decompiled share under each column</span>');
|
||||
drawLbMap();pmLoad().then(()=>{if(PM.doc)pmLayout()});renderLbWhere(s);renderLbTrend(LB.history);renderLbMilestones(LB.milestones);
|
||||
drawLbMap();pmLoad().then(()=>{if(PM.doc){pmLayout();renderLbFigs();renderLbWhere(s)}});renderLbWhere(s);renderLbTrend(LB.history);renderLbMilestones(LB.milestones);
|
||||
el('lbLogCount').textContent=LB.log_total?LB.log_total+' entries':'';
|
||||
put('lbLog',LB.log.length?LB.log.map(e=>'<li><time>'+esc(central(e.at))+'</time><div><div class="t"><span class="kind">'+esc(e.kind)+'</span>'+esc(e.title)+'</div><div class="d">'+esc(e.detail)+'</div></div></li>').join(''):'<li class="note">No entries yet.</li>')}
|
||||
put('lbLog',LB.log.length?LB.log.map(e=>'<li><time>'+esc(central(e.at))+'</time><div><div class="t"><span class="kind">'+esc(e.kind)+'</span>'+esc(e.title)+'</div><div class="d">'+esc(String(e.detail||'').startsWith(e.title)?String(e.detail).slice(e.title.length).replace(/^\n/,''):e.detail)+'</div></div></li>').join(''):'<li class="note">No entries yet.</li>')}
|
||||
async function renderLbWhere(s){const total=s.bytes||0,cat=k=>(s.by_category||{})[k]||{functions:0,bytes:0},uns=cat('unsorted'),sorted=total?100*(total-uns.bytes)/total:0;
|
||||
let tiers='';try{const p=await get('api/decomp/progress');tiers='<h3 class="sub-h" style="margin-top:18px">Game functions by how far they have come</h3>'+tierBar(p.tiers)+tierTable(p.tiers)}catch(e){}
|
||||
let tiers='';try{const p=PM.doc&&PM.doc.tiers?PM.doc:await get('api/decomp/progress');tiers='<h3 class="sub-h" style="margin-top:18px">Every function by how far it has come</h3><div class="note">Live from main. The game column is the one the headline counts; engine and platform functions that were decompiled anyway sit in the same tiers.</div>'+tierBar(p.tiers)+tierTable(p.tiers)}catch(e){}
|
||||
put('lbWhere','<div><span class="lb-big">'+pct(sorted)+'</span> <span class="note">of the executable’s code is sorted into a category · '+fmt(uns.functions)+' functions ('+kb(uns.bytes)+') still unsorted</span></div>'
|
||||
+'<div class="lb-stack" role="img" aria-label="Code bytes by category">'+LB_CATS.map(([k,n])=>{const sh=total?100*cat(k).bytes/total:0;return '<i class="'+(k==='unsorted'?'lb-hatch':'')+'" style="flex:0 0 '+sh.toFixed(3)+'%;background:var(--lb-'+k+')" title="'+esc(n)+': '+sh.toFixed(1)+'% of bytes"></i>'}).join('')+'</div>'
|
||||
+'<div class="lb-axis"><span>0</span><span>sorted '+pct(sorted)+' · unsorted '+pct(100-sorted)+'</span><span>'+kb(total)+'</span></div>'
|
||||
@@ -1227,7 +1234,8 @@ const kB=b=>b>=1e6?(b/1e6).toFixed(2)+' MB':(b/1e3).toFixed(2)+' kB';
|
||||
const hx=a=>'0x'+Number(a).toString(16).toUpperCase();
|
||||
const pc=(b,t)=>(t?100*b/t:0).toFixed(2);
|
||||
function rel(iso){const t=Date.parse(iso||'');if(!t)return 'at an unknown time';const s=(Date.now()-t)/1000;if(s<90)return 'just now';const m=s/60;if(m<90)return Math.round(m)+' minutes ago';const h=m/60;if(h<36)return Math.round(h)+' hours ago';return Math.round(h/24)+' days ago'}
|
||||
async function pmLoad(){if(PM.busy)return;PM.busy=true;try{
|
||||
function pmLoad(){if(!PM.pending)PM.pending=pmLoadOnce().finally(()=>{PM.pending=null});return PM.pending}
|
||||
async function pmLoadOnce(){if(PM.busy)return;PM.busy=true;try{
|
||||
const d=await get('api/logbook/progress-map'+(PM.version?'?have='+encodeURIComponent(PM.version):''));if(d.same)return;
|
||||
const U=d.unit_fields;d.units=d.units.map((u,i)=>{const o={i,fns:[]};U.forEach((k,j)=>o[k]=u[j]);o.pct=o.counted?100*(o.exact+o.verified+o.readable)/o.counted:null;o.lname=o.name.toLowerCase();return o});
|
||||
d.functions.forEach(f=>{const o={unit:f[0],address:f[1],size:f[2],status:d.statuses[f[3]],name:f[4]};o.lname=o.name.toLowerCase();d.units[o.unit].fns.push(o)});delete d.functions;
|
||||
@@ -1254,7 +1262,7 @@ function pmHeader(){const d=PM.doc;if(!d)return;const h=d.history,pt=PM.hist!=nu
|
||||
el('pmProject').textContent=d.project;put('pmFresh',pt?'':pmFresh(d.results));
|
||||
if(!pt){el('pmPct').textContent=d.percent.toFixed(2)+'%';
|
||||
put('pmBars',pmBar('Code',kB(c.bytes),pmCodeSegs(c,c.bytes),c.bytes,'game code · '+fmt(c.decompiled_functions)+' of '+fmt(c.functions)+' functions')
|
||||
+(d.data?pmBar('Data',fmt(d.data.addresses)+' addresses',[['pm-named',d.data.named,pc(d.data.named,d.data.addresses)+'% of the data addresses the code touches have a name in globals.csv ('+fmt(d.data.named)+' of '+fmt(d.data.addresses)+')']],d.data.addresses,'globals named'):''));
|
||||
+(d.all_code&&d.all_code.bytes?pmBar('All code',kB(d.all_code.bytes),[['pm-readable',d.all_code.decompiled_bytes||0,pc(d.all_code.decompiled_bytes||0,d.all_code.bytes)+'% of the whole executable decompiled ('+fmt(d.all_code.decompiled_functions||0)+' of '+fmt(d.all_code.functions)+' functions)']],d.all_code.bytes,'game + engine + platform · '+fmt(d.all_code.decompiled_functions||0)+' of '+fmt(d.all_code.functions)+' functions'):'')+(d.data?pmBar('Data',fmt(d.data.addresses)+' addresses',[['pm-named',d.data.named,pc(d.data.named,d.data.addresses)+'% of the data addresses the code touches have a name in globals.csv ('+fmt(d.data.named)+' of '+fmt(d.data.addresses)+')']],d.data.addresses,'globals named'):''));
|
||||
const at=d.commit.at||d.updated_at,moved=d.main.sha&&d.commit.sha&&d.main.sha.slice(0,7)!==String(d.commit.sha).slice(0,7);
|
||||
put('pmCommit','Updated <b title="'+esc(at||'')+'">'+esc(rel(at))+'</b> in commit <b class="mono">'+esc(d.commit.sha||'–')+'</b>'+(moved?' <span class="note">· main has moved on to '+esc(d.main.sha)+'; the statistics follow within minutes</span>':''));
|
||||
el('pmHistNote').textContent=d.percent!==d.logbook_percent&&d.logbook_percent!=null?'The Logbook statistics say '+d.logbook_percent+'%; they are being rebuilt.':''}
|
||||
@@ -1360,7 +1368,7 @@ const fmt=s=>esc(s).replace(/`([^`]+)`/g,'<code>$1</code>');
|
||||
const DAY_FMT=new Intl.DateTimeFormat('en-US',{timeZone:CENTRAL,year:'numeric',month:'numeric',day:'numeric'});
|
||||
const TIME_FMT=new Intl.DateTimeFormat('en-US',{timeZone:CENTRAL,hour:'numeric',minute:'2-digit',timeZoneName:'short'});
|
||||
const centralDay=d=>DAY_FMT.format(d);
|
||||
const centralTime=d=>TIME_FMT.format(d);
|
||||
const centralTime=d=>TIME_FMT.format(d).replace(/ C[DS]T$/,'')+' CST';
|
||||
const since=s=>{if(!s)return 'never';const m=Math.round((Date.now()-new Date(s))/60000);return m<1?'just now':m<120?m+' min ago':m<2880?Math.round(m/60)+' h ago':Math.round(m/1440)+' days ago'};
|
||||
let people=[],byName={},human='User',messages=[],lastId=0,filter='All',thread=null,sending=false,receipts={},overviewData=null,loaded=false;
|
||||
async function getJSON(path){const r=await fetch(path,{cache:'no-store'});const j=await r.json().catch(()=>({}));if(!r.ok)throw new Error(j.error||('HTTP '+r.status));return j}
|
||||
|
||||
@@ -58,6 +58,54 @@ class LogbookAuto(HarborCase):
|
||||
self.assertTrue(entry["at"].endswith("Z"))
|
||||
self.assertFalse(logbook_auto.refresh(self.h.project)["changed"])
|
||||
|
||||
def test_waits_for_the_lander_and_a_failed_stats_run_logs_nothing_twice(self):
|
||||
import os
|
||||
import time
|
||||
from unittest import mock
|
||||
logbook_auto._save_state(self.h.project, {"last_head": self.first})
|
||||
lock = self.dir / "lander-busy.lock"
|
||||
lock.write_text("x", encoding="utf-8")
|
||||
self.h.project.logbook.update(lander_lock=str(lock), stats_repo=str(self.dir / "stats"))
|
||||
out = logbook_auto.refresh(self.h.project)
|
||||
self.assertFalse(out["changed"])
|
||||
self.assertIn("lander", out["waiting"])
|
||||
self.assertEqual(logbook.read_all(self.h.project)["log"], [])
|
||||
old = time.time() - 61 * 60
|
||||
os.utime(lock, (old, old)) # stale: ignored
|
||||
with mock.patch.object(logbook_auto, "rebuild_stats", side_effect=RuntimeError("timed out")):
|
||||
out = logbook_auto.refresh(self.h.project)
|
||||
self.assertEqual((out["entries"], out["stats_error"]), (1, "timed out"))
|
||||
again = logbook_auto.refresh(self.h.project) # the statistics are retried ...
|
||||
self.assertIn("stats_error", again)
|
||||
self.assertNotIn("entries", again) # ... but the commit is not relogged
|
||||
self.assertEqual(len(logbook.read_all(self.h.project)["log"]), 1)
|
||||
|
||||
|
||||
class LanderGate(unittest.TestCase):
|
||||
def test_run_low_lowers_priority_and_ends_the_tree_on_timeout(self):
|
||||
import os
|
||||
import sys
|
||||
import time
|
||||
from harbor import lander_gate
|
||||
script = "import subprocess, sys, time; subprocess.Popen([sys.executable, '-c', 'import time; time.sleep(60)']);" \
|
||||
" print('ok', flush=True); time.sleep(60)"
|
||||
from unittest import mock
|
||||
seen = {}
|
||||
real = subprocess.Popen
|
||||
|
||||
def popen(argv, **kw):
|
||||
seen.update(kw)
|
||||
return real([sys.executable, "-c", "pass"], stdout=subprocess.PIPE, stderr=subprocess.PIPE, text=True)
|
||||
|
||||
with mock.patch.object(lander_gate.subprocess, "Popen", popen):
|
||||
self.assertEqual(lander_gate.run_low(["x"], 60).returncode, 0)
|
||||
if os.name == "nt":
|
||||
self.assertTrue(seen["creationflags"] & subprocess.BELOW_NORMAL_PRIORITY_CLASS)
|
||||
start = time.monotonic()
|
||||
with self.assertRaises(subprocess.TimeoutExpired):
|
||||
lander_gate.run_low([sys.executable, "-c", script], 3)
|
||||
self.assertLess(time.monotonic() - start, 45) # the grandchild did not hold the pipes
|
||||
|
||||
|
||||
class ProgressMapTests(HarborCase):
|
||||
"""GET /api/logbook/progress-map: units, tiers and the headline, by the Logbook's own definitions."""
|
||||
|
||||
@@ -208,6 +208,19 @@ class Refresher(HarborCase):
|
||||
rr.save_state(self.h.project, st)
|
||||
self.assertIsNone(rr.freshness(self.h.project)["match"]["sha"])
|
||||
|
||||
def test_waits_while_the_lander_runs_its_gates(self):
|
||||
lock = self.dir / "lander-busy.lock"
|
||||
self.h.project.logbook["lander_lock"] = str(lock)
|
||||
lock.write_text("lander", encoding="utf-8")
|
||||
os.utime(lock, (T0 - 300, T0 - 300))
|
||||
rr.request(self.h.project, "match")
|
||||
out = rr.tick(self.h.project, now=T0, hour=12)
|
||||
self.assertIn("lander", out["waiting"])
|
||||
self.assertEqual(self.calls, [])
|
||||
self.assertEqual(rr._requested(self.h.project), {"match"}) # kept for the next look
|
||||
self.assertTrue(rr.tick(self.h.project, now=T0 + 61 * 60, hour=12)["ok"]) # a lock over an hour old is stale
|
||||
self.assertEqual(len(self.calls), 1)
|
||||
|
||||
def test_runs_at_idle_priority(self):
|
||||
seen = {}
|
||||
|
||||
|
||||
@@ -58,6 +58,105 @@ class ReverseTools(unittest.TestCase):
|
||||
with self.assertRaises(HarborError):
|
||||
Reverse({})
|
||||
|
||||
def test_summary_names_status_vtables_and_hints(self):
|
||||
image = bytearray((self.d / "image.bin").read_bytes())
|
||||
image[0xC0:0xC8] = struct.pack("<II", 0x401000, 0x401040) # 0x4010C0: a two-slot vtable
|
||||
(self.d / "image.bin").write_bytes(bytes(image))
|
||||
(self.d / "sections.json").write_text(json.dumps([
|
||||
{"name": ".text", "start": 0x401000, "end": 0x401080}, {"name": ".rdata", "start": 0x4010B0,
|
||||
"end": 0x401100}]), encoding="utf-8")
|
||||
facts = [json.loads(l) for l in (self.d / "facts.jsonl").read_text(encoding="utf-8").splitlines()]
|
||||
facts[1]["data"] = ["0x4010C0"] # the constructor installs it
|
||||
facts[0]["strings"] = [".cue"]
|
||||
(self.d / "facts.jsonl").write_text("\n".join(json.dumps(f) for f in facts), encoding="utf-8")
|
||||
src = self.d / "src" / "game"
|
||||
src.mkdir(parents=True)
|
||||
(src / "401040_Helper.c").write_text("/* Helper\n * status: readable\n */\nvoid Helper(void) {}\n",
|
||||
encoding="utf-8")
|
||||
(self.d / "mac.csv").write_text("pc_address,demangled,confidence\n0x00401000,Level::Get(),high\n",
|
||||
encoding="utf-8")
|
||||
r = Reverse({"image": str(self.d / "image.bin"), "base": "0x401000", "facts": str(self.d / "facts.jsonl"),
|
||||
"annotations": str(self.d / "annotations"), "src": str(self.d / "src"),
|
||||
"mac_names": str(self.d / "mac.csv")})
|
||||
s = r.summary("GetLevel")
|
||||
self.assertEqual((s["name"], s["size"], s["status"], s["readable"]), ("GetLevel", 18, "no file", False))
|
||||
self.assertIn("plain ret", s["prototype"])
|
||||
self.assertEqual(s["callees"], [{"address": "0x401040", "name": "Helper", "status": "readable",
|
||||
"readable": True}])
|
||||
self.assertEqual(s["globals"], [{"address": "0x401080", "name": "g_root", "type": "int"}])
|
||||
self.assertEqual(s["strings"], [".cue"])
|
||||
self.assertEqual(s["vtable_slots"], [{"table": "0x4010C0", "slot": 0, "offset": "0x0", "kind": "vtable"}])
|
||||
self.assertEqual(s["hints"]["mac_name"]["demangled"], "Level::Get()")
|
||||
h = r.summary("0x401040")
|
||||
self.assertTrue(h["readable"])
|
||||
self.assertEqual([c["name"] for c in h["callers"]], ["GetLevel"])
|
||||
self.assertEqual(h["vtable_slots"][0]["slot"], 1)
|
||||
from harbor.reverse import format_summary
|
||||
self.assertIn("vtable 0x4010C0 slot 1 (+0x4)", format_summary(h))
|
||||
|
||||
|
||||
class GhidraCheck(unittest.TestCase):
|
||||
"""ghidra_check compares the served program with the project image: range, SHA-256 of the file, byte samples."""
|
||||
|
||||
def serve(self, lo, hi, path, image):
|
||||
import http.server
|
||||
import threading
|
||||
|
||||
class Fake(http.server.BaseHTTPRequestHandler):
|
||||
def log_message(self, *a):
|
||||
pass
|
||||
|
||||
def do_POST(self):
|
||||
body = json.loads(self.rfile.read(int(self.headers["Content-Length"])))
|
||||
if "id" not in body:
|
||||
self.send_response(202)
|
||||
self.end_headers()
|
||||
return
|
||||
params = body.get("params", {})
|
||||
name, args = params.get("name"), params.get("arguments", {})
|
||||
text = ""
|
||||
if name == "get_binary_info":
|
||||
text = f"Program Information:\nName: image.bin\nImage Base: 00000000\nMin Address: {lo:08x}\n" \
|
||||
f"Max Address: {hi:08x}\nFunction Count: 2\n"
|
||||
elif name == "list_binaries":
|
||||
text = f"Open Programs in Ghidra:\n\n1. image.bin [ACTIVE]\n Executable Path: /{path}\n"
|
||||
elif name == "get_data_at":
|
||||
a, n = int(args["address"], 16), int(args["len"])
|
||||
chunk = image[a - 0x401000:a - 0x401000 + n]
|
||||
rows = [f"{a + i:x} " + " ".join(f"{b:02x}" for b in chunk[i:i + 16]) + " |....|"
|
||||
for i in range(0, len(chunk), 16)]
|
||||
text = f"Hexdump at {a:08x} ({n} bytes):\n\n" + "\n".join(rows) + "\n"
|
||||
result = {"content": [{"type": "text", "text": text}]} if body["method"] == "tools/call" else {}
|
||||
data = json.dumps({"jsonrpc": "2.0", "id": body["id"], "result": result}).encode()
|
||||
self.send_response(200)
|
||||
self.send_header("Content-Type", "application/json")
|
||||
self.send_header("Content-Length", str(len(data)))
|
||||
self.end_headers()
|
||||
self.wfile.write(data)
|
||||
|
||||
server = http.server.ThreadingHTTPServer(("127.0.0.1", 0), Fake)
|
||||
threading.Thread(target=server.serve_forever, daemon=True).start()
|
||||
self.addCleanup(server.server_close)
|
||||
self.addCleanup(server.shutdown)
|
||||
return f"http://127.0.0.1:{server.server_port}/mcp"
|
||||
|
||||
def test_match_and_mismatch(self):
|
||||
from harbor.reverse import ghidra_check
|
||||
d = temp_dir()
|
||||
self.addCleanup(shutil.rmtree, d, True)
|
||||
image = bytes(range(256)) * 4
|
||||
(d / "image.bin").write_bytes(image)
|
||||
config = {"image": str(d / "image.bin"), "base": "0x401000"}
|
||||
good = ghidra_check(config | {"ghidra_url": self.serve(0x401000, 0x4013FF, (d / "image.bin").as_posix(), image)})
|
||||
self.assertTrue(good["ok"], good)
|
||||
self.assertEqual([c["ok"] for c in good["checks"]], [True, True, True])
|
||||
other = d / "other.bin"
|
||||
other.write_bytes(image[:-1] + b"\x00")
|
||||
bad = ghidra_check(config | {"ghidra_url": self.serve(0x401000, 0x4013FE, other.as_posix(),
|
||||
image[:512] + b"\0" * 512)})
|
||||
self.assertFalse(bad["ok"])
|
||||
self.assertEqual([c["ok"] for c in bad["checks"]], [False, False, False])
|
||||
|
||||
|
||||
class GhidraForwarder(unittest.TestCase):
|
||||
"""The ghidra() tool against a fake streamable-HTTP MCP server: a session, a polled async task, a refusal."""
|
||||
|
||||
+15
-2
@@ -18,6 +18,19 @@ sys.path.insert(0, str(Path(__file__).resolve().parent.parent))
|
||||
from harbor import projects # noqa: E402
|
||||
|
||||
|
||||
def cst(iso):
|
||||
"""'2026-10-07T23:21:05Z' -> '18:21 CST' (US Central, as the User reads every time)."""
|
||||
import datetime
|
||||
from zoneinfo import ZoneInfo
|
||||
try:
|
||||
when = datetime.datetime.fromisoformat(str(iso).replace("Z", "+00:00"))
|
||||
except ValueError:
|
||||
return str(iso)
|
||||
if when.tzinfo is None:
|
||||
when = when.replace(tzinfo=datetime.timezone.utc)
|
||||
return when.astimezone(ZoneInfo("America/Chicago")).strftime("%H:%M") + " CST"
|
||||
|
||||
|
||||
def snapshot(conn, me):
|
||||
q = lambda sql, *a: conn.execute(sql, a).fetchall()
|
||||
return {
|
||||
@@ -88,7 +101,7 @@ def main():
|
||||
for agent, last in stalled(conn, project, args.me, args.stall):
|
||||
if agent in {a.strip() for a in args.ignore.split(",") if a.strip()}:
|
||||
continue
|
||||
found.append(f"stalled: {agent} has made no tool call since {last[11:16]}Z (not stopped) - wake it")
|
||||
found.append(f"stalled: {agent} has made no tool call since {cst(last)} (not stopped) - wake it")
|
||||
if found:
|
||||
print("\n".join(found))
|
||||
return 0
|
||||
@@ -96,7 +109,7 @@ def main():
|
||||
tasks = conn.execute("SELECT state, COUNT(*) FROM tasks GROUP BY state").fetchall()
|
||||
last = conn.execute("SELECT agent, MAX(at) FROM activity_log GROUP BY agent").fetchall()
|
||||
print(f"quiet for {args.minutes:g} min; open review items {open_items}; tasks {dict(tasks)}; last tool call "
|
||||
+ ", ".join(f"{a} {t[11:16]}Z" for a, t in last))
|
||||
+ ", ".join(f"{a} {cst(t)}" for a, t in last))
|
||||
return 0
|
||||
|
||||
|
||||
|
||||
Reference in new issue
Block a user