From aa20a5941f675ce040488ee4d57d99b5abc42fef Mon Sep 17 00:00:00 2001 From: Theo Weijmer Date: Thu, 1 Oct 2026 09:35:24 +0200 Subject: [PATCH] Add dictation skeleton implementation plan Co-Authored-By: Claude Sonnet 5.5 --- .../plans/2026-10-01-dictation-skeleton.md | 1399 +++++++++++++++++ 1 file changed, 1399 insertions(+) create mode 100644 docs/superpowers/plans/2026-10-01-dictation-skeleton.md diff --git a/docs/superpowers/plans/2026-10-01-dictation-skeleton.md b/docs/superpowers/plans/2026-10-01-dictation-skeleton.md new file mode 100644 index 0000000..f256094 --- /dev/null +++ b/docs/superpowers/plans/2026-10-01-dictation-skeleton.md @@ -0,0 +1,1399 @@ +# Dictatie-skeleton Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** Een werkende macOS-menubalk-app: Fn ingedrukt houden neemt op, loslaten transcribeert lokaal (NL/EN) en plakt de tekst op de cursor. + +**Architecture:** Een Swift Package. `SstCore` bevat de logica achter protocollen (testbaar met fakes), `SstWhisper` bevat de WhisperKit-implementatie, `Sst` is de SwiftUI-menubalk-app en `SstProbe` is een CLI om hotkey, opname en transcriptie los te testen. Een script bouwt de package tot een `.app`-bundle. + +**Tech Stack:** Swift 6.4 (language mode 5), SwiftUI `MenuBarExtra`, `AVAudioEngine`, `CGEventTap`, WhisperKit, Swift Testing. + +**Spec:** `docs/superpowers/specs/2026-10-01-push-to-talk-dictation-design.md` + +## Global Constraints + +- Alleen Apple Silicon, macOS 14+ als deployment target (doelmachine: M4 Pro, macOS 27, Xcode 27). +- Alles lokaal: geen netwerkverkeer behalve de eenmalige modeldownload; audio wordt nooit opgeslagen. +- Audioformaat naar de transcriber: 16 kHz, mono, Float32. +- Opnames korter dan 0,3 s (4800 samples) worden genegeerd. +- Model: Whisper `large-v3-turbo` via WhisperKit, automatische taalherkenning (Nederlands en Engels). +- Niet-gesandboxed; vereiste rechten: Microfoon, Accessibility, Input Monitoring. +- Statemachine: `idle → recording → transcribing → inserting → idle`; fouten gaan naar `onError` en eindigen in `idle`. +- Falend invoegen laat de tekst op het klembord staan. +- Elke commit eindigt met de trailer `Co-Authored-By: Claude Sonnet 5.5 `. + +**Afwijkingen van de spec (bewust):** +- De "dunne Xcode-app" is een app-bundle die `scripts/make-app.sh` uit de package assembleert. Een handgeschreven `.xcodeproj` is fragiel; Xcode kan `Package.swift` rechtstreeks openen. +- Swift language mode 5, om strikte Sendable-regels buiten de skeleton-scope te houden. +- De overlay-pill (bouwvolgorde-stap 5) valt buiten dit plan; de menubalk-icoon toont de status. + +## Bestandsstructuur + +``` +Package.swift +.gitignore +scripts/make-app.sh +Sources/SstCore/Models.swift HotkeyEvent, DictationState, CleaningContext, protocollen +Sources/SstCore/DictationController.swift statemachine +Sources/SstCore/PassthroughCleaner.swift +Sources/SstCore/TextInserter.swift TextInserter + PasteboardProviding/KeystrokePosting +Sources/SstCore/SystemPasteboard.swift NSPasteboard + CGEvent-implementaties +Sources/SstCore/FnKeyTracker.swift pure Fn-logica +Sources/SstCore/HotkeyMonitor.swift CGEventTap +Sources/SstCore/AudioRecorder.swift AVAudioEngine → 16 kHz mono +Sources/SstWhisper/WhisperKitTranscriber.swift +Sources/SstProbe/main.swift, HotkeyProbe.swift, RecordProbe.swift +Sources/Sst/SstApp.swift, AppModel.swift, MenuContent.swift +Tests/SstCoreTests/Fakes.swift, DictationControllerTests.swift, + TextInserterTests.swift, FnKeyTrackerTests.swift +``` + +--- + +### Task 1: Package-scaffold en DictationController + +**Files:** +- Create: `Package.swift`, `.gitignore` +- Create: `Sources/SstCore/Models.swift`, `Sources/SstCore/DictationController.swift` +- Test: `Tests/SstCoreTests/Fakes.swift`, `Tests/SstCoreTests/DictationControllerTests.swift` + +**Interfaces:** +- Produces (gebruikt door alle latere taken): + - `enum HotkeyEvent { case pressed, released }` + - `enum DictationState: Equatable { case idle, recording, transcribing, inserting }` + - `struct CleaningContext { init() }` + - `protocol AudioRecording { func start() throws; func stop() -> [Float] }` + - `protocol Transcriber { func transcribe(_ samples: [Float]) async throws -> String }` + - `protocol Cleaner { func clean(_ text: String, context: CleaningContext) async throws -> String }` + - `protocol TextInserting { func insert(_ text: String) async throws }` + - `@MainActor final class DictationController` met `init(recorder:transcriber:cleaner:inserter:minimumSamples:)`, `var onStateChange: ((DictationState) -> Void)?`, `var onError: ((String) -> Void)?`, `func handle(_ event: HotkeyEvent) async` + +- [ ] **Step 1: Scaffold maken** + +`Package.swift`: + +```swift +// swift-tools-version: 6.0 +import PackageDescription + +let package = Package( + name: "Sst", + platforms: [.macOS(.v14)], + targets: [ + .target(name: "SstCore"), + .testTarget(name: "SstCoreTests", dependencies: ["SstCore"]), + ], + swiftLanguageModes: [.v5] +) +``` + +`.gitignore`: + +``` +.build/ +build/ +.DS_Store +*.xcodeproj +``` + +`Sources/SstCore/Models.swift`: + +```swift +import Foundation + +public enum HotkeyEvent: Equatable, Sendable { + case pressed + case released +} + +public enum DictationState: Equatable, Sendable { + case idle + case recording + case transcribing + case inserting +} + +public struct CleaningContext: Sendable { + public init() {} +} + +public protocol AudioRecording { + func start() throws + /// Stops recording and returns all captured 16 kHz mono Float samples. + func stop() -> [Float] +} + +public protocol Transcriber { + func transcribe(_ samples: [Float]) async throws -> String +} + +public protocol Cleaner { + func clean(_ text: String, context: CleaningContext) async throws -> String +} + +public protocol TextInserting { + func insert(_ text: String) async throws +} +``` + +- [ ] **Step 2: Fakes en falende tests schrijven** + +`Tests/SstCoreTests/Fakes.swift`: + +```swift +import Foundation +@testable import SstCore + +struct TestError: Error {} + +final class FakeRecorder: AudioRecording { + var samples: [Float] = [] + var startError: Error? + private(set) var startCount = 0 + + func start() throws { + startCount += 1 + if let startError { throw startError } + } + + func stop() -> [Float] { samples } +} + +final class FakeTranscriber: Transcriber { + var result: Result = .success("hallo wereld") + private(set) var callCount = 0 + + func transcribe(_ samples: [Float]) async throws -> String { + callCount += 1 + return try result.get() + } +} + +final class FakeCleaner: Cleaner { + func clean(_ text: String, context: CleaningContext) async throws -> String { + "[\(text)]" + } +} + +final class FakeInserter: TextInserting { + var error: Error? + private(set) var inserted: [String] = [] + + func insert(_ text: String) async throws { + if let error { throw error } + inserted.append(text) + } +} +``` + +`Tests/SstCoreTests/DictationControllerTests.swift`: + +```swift +import Testing +@testable import SstCore + +@MainActor +struct DictationControllerTests { + let recorder = FakeRecorder() + let transcriber = FakeTranscriber() + let cleaner = FakeCleaner() + let inserter = FakeInserter() + static let oneSecond = [Float](repeating: 0.1, count: 16_000) + + func makeController() -> DictationController { + DictationController( + recorder: recorder, + transcriber: transcriber, + cleaner: cleaner, + inserter: inserter + ) + } + + @Test func pressAndReleaseTranscribesCleansAndInserts() async { + recorder.samples = Self.oneSecond + let controller = makeController() + var states: [DictationState] = [] + controller.onStateChange = { states.append($0) } + + await controller.handle(.pressed) + await controller.handle(.released) + + #expect(states == [.recording, .transcribing, .inserting, .idle]) + #expect(inserter.inserted == ["[hallo wereld]"]) + } + + @Test func shortRecordingIsDiscarded() async { + recorder.samples = [Float](repeating: 0.1, count: 1_000) + let controller = makeController() + var states: [DictationState] = [] + controller.onStateChange = { states.append($0) } + + await controller.handle(.pressed) + await controller.handle(.released) + + #expect(states == [.recording, .idle]) + #expect(transcriber.callCount == 0) + #expect(inserter.inserted.isEmpty) + } + + @Test func emptyTranscriptionIsNotInserted() async { + recorder.samples = Self.oneSecond + transcriber.result = .success(" \n") + let controller = makeController() + var errors: [String] = [] + controller.onError = { errors.append($0) } + + await controller.handle(.pressed) + await controller.handle(.released) + + #expect(inserter.inserted.isEmpty) + #expect(errors.isEmpty) + #expect(controller.state == .idle) + } + + @Test func transcriberErrorReportsAndReturnsToIdle() async { + recorder.samples = Self.oneSecond + transcriber.result = .failure(TestError()) + let controller = makeController() + var errors: [String] = [] + controller.onError = { errors.append($0) } + + await controller.handle(.pressed) + await controller.handle(.released) + + #expect(errors.count == 1) + #expect(inserter.inserted.isEmpty) + #expect(controller.state == .idle) + } + + @Test func inserterErrorReportsAndReturnsToIdle() async { + recorder.samples = Self.oneSecond + inserter.error = TestError() + let controller = makeController() + var errors: [String] = [] + controller.onError = { errors.append($0) } + + await controller.handle(.pressed) + await controller.handle(.released) + + #expect(errors.count == 1) + #expect(controller.state == .idle) + } + + @Test func recorderStartFailureStaysIdle() async { + recorder.startError = TestError() + let controller = makeController() + var errors: [String] = [] + controller.onError = { errors.append($0) } + + await controller.handle(.pressed) + + #expect(errors.count == 1) + #expect(controller.state == .idle) + } + + @Test func secondPressWhileRecordingIsIgnored() async { + let controller = makeController() + await controller.handle(.pressed) + await controller.handle(.pressed) + #expect(recorder.startCount == 1) + } + + @Test func releaseWithoutPressIsIgnored() async { + let controller = makeController() + var states: [DictationState] = [] + controller.onStateChange = { states.append($0) } + + await controller.handle(.released) + + #expect(states.isEmpty) + #expect(transcriber.callCount == 0) + } +} +``` + +- [ ] **Step 3: Test draaien, moet falen** + +Run: `swift test --filter DictationControllerTests` +Expected: compilatiefout `cannot find 'DictationController' in scope`. + +- [ ] **Step 4: Minimale implementatie** + +`Sources/SstCore/DictationController.swift`: + +```swift +import Foundation + +@MainActor +public final class DictationController { + public private(set) var state: DictationState = .idle { + didSet { onStateChange?(state) } + } + public var onStateChange: ((DictationState) -> Void)? + public var onError: ((String) -> Void)? + + private let recorder: AudioRecording + private let transcriber: Transcriber + private let cleaner: Cleaner + private let inserter: TextInserting + private let minimumSamples: Int + + /// - Parameter minimumSamples: 16 kHz samples; default is 0.3 s. + public init( + recorder: AudioRecording, + transcriber: Transcriber, + cleaner: Cleaner, + inserter: TextInserting, + minimumSamples: Int = 4_800 + ) { + self.recorder = recorder + self.transcriber = transcriber + self.cleaner = cleaner + self.inserter = inserter + self.minimumSamples = minimumSamples + } + + public func handle(_ event: HotkeyEvent) async { + switch (event, state) { + case (.pressed, .idle): + do { + try recorder.start() + state = .recording + } catch { + onError?(error.localizedDescription) + } + case (.released, .recording): + let samples = recorder.stop() + guard samples.count >= minimumSamples else { + state = .idle + return + } + await process(samples) + default: + break + } + } + + private func process(_ samples: [Float]) async { + defer { state = .idle } + do { + state = .transcribing + let raw = try await transcriber.transcribe(samples) + let text = raw.trimmingCharacters(in: .whitespacesAndNewlines) + guard !text.isEmpty else { return } + let cleaned = try await cleaner.clean(text, context: CleaningContext()) + state = .inserting + try await inserter.insert(cleaned) + } catch { + onError?(error.localizedDescription) + } + } +} +``` + +- [ ] **Step 5: Test draaien, moet slagen** + +Run: `swift test --filter DictationControllerTests` +Expected: 8 tests PASS. + +- [ ] **Step 6: Commit** + +```bash +git add Package.swift .gitignore Sources Tests +git commit -m "feat: add package scaffold and DictationController state machine" -m "Co-Authored-By: Claude Sonnet 5.5 " +``` + +--- + +### Task 2: PassthroughCleaner en TextInserter + +**Files:** +- Create: `Sources/SstCore/PassthroughCleaner.swift`, `Sources/SstCore/TextInserter.swift`, `Sources/SstCore/SystemPasteboard.swift` +- Test: `Tests/SstCoreTests/TextInserterTests.swift` + +**Interfaces:** +- Consumes: `TextInserting`, `Cleaner`, `CleaningContext` uit Task 1. +- Produces: + - `typealias PasteboardItem = [String: Data]` + - `protocol PasteboardProviding { func readItems() -> [PasteboardItem]; func writeString(_ text: String); func writeItems(_ items: [PasteboardItem]) }` + - `protocol KeystrokePosting { func postPaste() throws }` + - `struct TextInserter: TextInserting`, `init(pasteboard:paster:restoreDelay: Duration = .milliseconds(150))` + - `struct PassthroughCleaner: Cleaner` + - `struct SystemPasteboard: PasteboardProviding`, `struct CGEventPaster: KeystrokePosting` (init zonder argumenten) + +- [ ] **Step 1: Falende tests schrijven** + +`Tests/SstCoreTests/TextInserterTests.swift`: + +```swift +import Foundation +import Testing +@testable import SstCore + +final class FakePasteboard: PasteboardProviding { + var items: [PasteboardItem] = [] + + func readItems() -> [PasteboardItem] { items } + func writeString(_ text: String) { + items = [["public.utf8-plain-text": Data(text.utf8)]] + } + func writeItems(_ items: [PasteboardItem]) { self.items = items } +} + +final class FakePaster: KeystrokePosting { + var shouldFail = false + var onPaste: (() -> Void)? + private(set) var pasteCount = 0 + + func postPaste() throws { + pasteCount += 1 + if shouldFail { throw TestError() } + onPaste?() + } +} + +struct TextInserterTests { + let pasteboard = FakePasteboard() + let paster = FakePaster() + + func makeInserter() -> TextInserter { + TextInserter(pasteboard: pasteboard, paster: paster, restoreDelay: .zero) + } + + @Test func pastesTextThenRestoresPreviousClipboard() async throws { + let original: [PasteboardItem] = [["public.utf8-plain-text": Data("oud".utf8)]] + pasteboard.items = original + var seenAtPaste: Data? + paster.onPaste = { seenAtPaste = pasteboard.items.first?["public.utf8-plain-text"] } + + try await makeInserter().insert("hallo") + + #expect(seenAtPaste == Data("hallo".utf8)) + #expect(paster.pasteCount == 1) + #expect(pasteboard.items == original) + } + + @Test func failedPasteLeavesTextOnClipboard() async { + pasteboard.items = [["public.utf8-plain-text": Data("oud".utf8)]] + paster.shouldFail = true + + await #expect(throws: TestError.self) { + try await makeInserter().insert("hallo") + } + + #expect(pasteboard.items.first?["public.utf8-plain-text"] == Data("hallo".utf8)) + } + + @Test func passthroughCleanerReturnsTextUnchanged() async throws { + let result = try await PassthroughCleaner().clean("hoi", context: CleaningContext()) + #expect(result == "hoi") + } +} +``` + +- [ ] **Step 2: Test draaien, moet falen** + +Run: `swift test --filter TextInserterTests` +Expected: compilatiefout `cannot find type 'PasteboardProviding' in scope`. + +- [ ] **Step 3: Implementatie** + +`Sources/SstCore/PassthroughCleaner.swift`: + +```swift +public struct PassthroughCleaner: Cleaner { + public init() {} + + public func clean(_ text: String, context: CleaningContext) async throws -> String { + text + } +} +``` + +`Sources/SstCore/TextInserter.swift`: + +```swift +import Foundation + +public typealias PasteboardItem = [String: Data] + +public protocol PasteboardProviding { + func readItems() -> [PasteboardItem] + func writeString(_ text: String) + func writeItems(_ items: [PasteboardItem]) +} + +public protocol KeystrokePosting { + func postPaste() throws +} + +public struct TextInserter: TextInserting { + private let pasteboard: PasteboardProviding + private let paster: KeystrokePosting + private let restoreDelay: Duration + + public init( + pasteboard: PasteboardProviding, + paster: KeystrokePosting, + restoreDelay: Duration = .milliseconds(150) + ) { + self.pasteboard = pasteboard + self.paster = paster + self.restoreDelay = restoreDelay + } + + public func insert(_ text: String) async throws { + let saved = pasteboard.readItems() + pasteboard.writeString(text) + // On failure the text deliberately stays on the clipboard. + try paster.postPaste() + // Give the target app time to read the clipboard before restoring it. + try await Task.sleep(for: restoreDelay) + pasteboard.writeItems(saved) + } +} +``` + +`Sources/SstCore/SystemPasteboard.swift`: + +```swift +import AppKit +import ApplicationServices + +public struct SystemPasteboard: PasteboardProviding { + public init() {} + + public func readItems() -> [PasteboardItem] { + (NSPasteboard.general.pasteboardItems ?? []).map { item in + var dict: PasteboardItem = [:] + for type in item.types { + if let data = item.data(forType: type) { dict[type.rawValue] = data } + } + return dict + } + } + + public func writeString(_ text: String) { + let pasteboard = NSPasteboard.general + pasteboard.clearContents() + pasteboard.setString(text, forType: .string) + } + + public func writeItems(_ items: [PasteboardItem]) { + let pasteboard = NSPasteboard.general + pasteboard.clearContents() + let restored = items.map { dict -> NSPasteboardItem in + let item = NSPasteboardItem() + for (type, data) in dict { item.setData(data, forType: NSPasteboard.PasteboardType(type)) } + return item + } + if !restored.isEmpty { pasteboard.writeObjects(restored) } + } +} + +public struct CGEventPaster: KeystrokePosting { + public enum PasteError: Error, LocalizedError { + case accessibilityDenied + case eventCreationFailed + + public var errorDescription: String? { + switch self { + case .accessibilityDenied: + "Geen Accessibility-rechten: tekst staat op het klembord, plak zelf met ⌘V." + case .eventCreationFailed: + "Kon het plak-event niet aanmaken: tekst staat op het klembord." + } + } + } + + public init() {} + + public func postPaste() throws { + guard AXIsProcessTrusted() else { throw PasteError.accessibilityDenied } + let source = CGEventSource(stateID: .combinedSessionState) + let vKey: CGKeyCode = 0x09 + guard + let down = CGEvent(keyboardEventSource: source, virtualKey: vKey, keyDown: true), + let up = CGEvent(keyboardEventSource: source, virtualKey: vKey, keyDown: false) + else { throw PasteError.eventCreationFailed } + down.flags = .maskCommand + up.flags = .maskCommand + down.post(tap: .cgAnnotatedSessionEventTap) + up.post(tap: .cgAnnotatedSessionEventTap) + } +} +``` + +- [ ] **Step 4: Alle tests draaien** + +Run: `swift test` +Expected: alle tests PASS (8 controller + 3 inserter/cleaner). + +- [ ] **Step 5: Commit** + +```bash +git add Sources Tests +git commit -m "feat: add TextInserter with clipboard restore and PassthroughCleaner" -m "Co-Authored-By: Claude Sonnet 5.5 " +``` + +--- + +### Task 3: Fn-hotkey (FnKeyTracker, HotkeyMonitor) en probe + +**Files:** +- Create: `Sources/SstCore/FnKeyTracker.swift`, `Sources/SstCore/HotkeyMonitor.swift` +- Create: `Sources/SstProbe/main.swift`, `Sources/SstProbe/HotkeyProbe.swift` +- Modify: `Package.swift` +- Test: `Tests/SstCoreTests/FnKeyTrackerTests.swift` + +**Interfaces:** +- Consumes: `HotkeyEvent` uit Task 1. +- Produces: + - `struct FnKeyTracker { init(); mutating func process(fnDown: Bool) -> HotkeyEvent? }` + - `final class HotkeyMonitor { init(); var onEvent: ((HotkeyEvent) -> Void)?; func start() throws; func stop() }`, `HotkeyMonitor.HotkeyError: Error, LocalizedError` + - Executable `SstProbe` met subcommando `hotkey`. + +- [ ] **Step 1: Falende test schrijven** + +`Tests/SstCoreTests/FnKeyTrackerTests.swift`: + +```swift +import Testing +@testable import SstCore + +struct FnKeyTrackerTests { + @Test func emitsPressedThenReleasedOnlyOnTransitions() { + var tracker = FnKeyTracker() + #expect(tracker.process(fnDown: false) == nil) + #expect(tracker.process(fnDown: true) == .pressed) + #expect(tracker.process(fnDown: true) == nil) + #expect(tracker.process(fnDown: false) == .released) + #expect(tracker.process(fnDown: false) == nil) + } +} +``` + +- [ ] **Step 2: Test draaien, moet falen** + +Run: `swift test --filter FnKeyTrackerTests` +Expected: compilatiefout `cannot find 'FnKeyTracker' in scope`. + +- [ ] **Step 3: Implementatie** + +`Sources/SstCore/FnKeyTracker.swift`: + +```swift +public struct FnKeyTracker { + private var isDown = false + + public init() {} + + /// Feed the current Fn flag state; returns an event only on a transition. + public mutating func process(fnDown: Bool) -> HotkeyEvent? { + guard fnDown != isDown else { return nil } + isDown = fnDown + return fnDown ? .pressed : .released + } +} +``` + +`Sources/SstCore/HotkeyMonitor.swift`: + +```swift +import ApplicationServices +import CoreGraphics + +public final class HotkeyMonitor { + public enum HotkeyError: Error, LocalizedError { + case tapUnavailable + + public var errorDescription: String? { + "Kan de Fn-toets niet volgen: geef Input Monitoring-rechten in Systeeminstellingen en probeer opnieuw (soms is een herstart van de app nodig)." + } + } + + public var onEvent: ((HotkeyEvent) -> Void)? + + private var tracker = FnKeyTracker() + private var tap: CFMachPort? + private var source: CFRunLoopSource? + + public init() {} + + public func start() throws { + guard tap == nil else { return } + let mask = CGEventMask(1 << CGEventType.flagsChanged.rawValue) + let callback: CGEventTapCallBack = { _, type, event, refcon in + if let refcon { + let monitor = Unmanaged.fromOpaque(refcon).takeUnretainedValue() + monitor.handle(type: type, event: event) + } + return Unmanaged.passUnretained(event) + } + guard let tap = CGEvent.tapCreate( + tap: .cgSessionEventTap, + place: .headInsertEventTap, + options: .listenOnly, + eventsOfInterest: mask, + callback: callback, + userInfo: Unmanaged.passUnretained(self).toOpaque() + ) else { throw HotkeyError.tapUnavailable } + + let source = CFMachPortCreateRunLoopSource(kCFAllocatorDefault, tap, 0) + CFRunLoopAddSource(CFRunLoopGetMain(), source, .commonModes) + CGEvent.tapEnable(tap: tap, enable: true) + self.tap = tap + self.source = source + } + + public func stop() { + if let source { CFRunLoopRemoveSource(CFRunLoopGetMain(), source, .commonModes) } + if let tap { CGEvent.tapEnable(tap: tap, enable: false) } + tap = nil + source = nil + } + + private func handle(type: CGEventType, event: CGEvent) { + switch type { + case .tapDisabledByTimeout, .tapDisabledByUserInput: + if let tap { CGEvent.tapEnable(tap: tap, enable: true) } + case .flagsChanged: + let fnDown = event.flags.contains(.maskSecondaryFn) + if let hotkeyEvent = tracker.process(fnDown: fnDown) { + onEvent?(hotkeyEvent) + } + default: + break + } + } +} +``` + +`Package.swift` (vervang volledig): + +```swift +// swift-tools-version: 6.0 +import PackageDescription + +let package = Package( + name: "Sst", + platforms: [.macOS(.v14)], + targets: [ + .target(name: "SstCore"), + .executableTarget(name: "SstProbe", dependencies: ["SstCore"]), + .testTarget(name: "SstCoreTests", dependencies: ["SstCore"]), + ], + swiftLanguageModes: [.v5] +) +``` + +`Sources/SstProbe/main.swift`: + +```swift +import Foundation + +let command = CommandLine.arguments.dropFirst().first ?? "help" +switch command { +case "hotkey": + runHotkeyProbe() +default: + print("usage: sst-probe hotkey") +} +``` + +`Sources/SstProbe/HotkeyProbe.swift`: + +```swift +import Foundation +import CoreGraphics +import SstCore + +func runHotkeyProbe() -> Never { + if !CGPreflightListenEventAccess() { _ = CGRequestListenEventAccess() } + let monitor = HotkeyMonitor() + monitor.onEvent = { print("hotkey:", $0) } + do { try monitor.start() } catch { + print("failed:", error.localizedDescription) + exit(1) + } + print("Houd Fn ingedrukt en laat los. Ctrl-C om te stoppen.") + RunLoop.main.run() + exit(0) +} +``` + +- [ ] **Step 4: Tests draaien** + +Run: `swift test` +Expected: alle tests PASS (incl. `FnKeyTrackerTests`). + +- [ ] **Step 5: Handmatige check van de hotkey** + +Zet eerst in Systeeminstellingen → Toetsenbord "Druk op 🌐 voor" op **Doe niets** (anders opent Fn de emoji-kiezer/dicteren). + +Run: `swift run SstProbe hotkey` +Geef de terminal-app (waar je dit draait) Input Monitoring-rechten als daarom gevraagd wordt en start opnieuw. +Expected: bij Fn omlaag `hotkey: pressed`, bij loslaten `hotkey: released`, geen dubbele events. + +- [ ] **Step 6: Commit** + +```bash +git add Package.swift Sources Tests +git commit -m "feat: add Fn hotkey monitor and probe CLI" -m "Co-Authored-By: Claude Sonnet 5.5 " +``` + +--- + +### Task 4: AudioRecorder en record-probe + +**Files:** +- Create: `Sources/SstCore/AudioRecorder.swift`, `Sources/SstProbe/RecordProbe.swift` +- Modify: `Sources/SstProbe/main.swift` + +**Interfaces:** +- Consumes: `AudioRecording` uit Task 1. +- Produces: + - `final class AudioRecorder: AudioRecording`, `init()`, `AudioRecorder.RecorderError: Error, LocalizedError` + - Probe-helper `func recordSamples(seconds: Double) async -> [Float]?` (hergebruikt in Task 5) + - Probe-subcommando `record` + +De AudioRecorder is hardware-afhankelijk en wordt daarom handmatig getest via de probe. + +- [ ] **Step 1: Implementatie** + +`Sources/SstCore/AudioRecorder.swift`: + +```swift +import AVFoundation + +public final class AudioRecorder: AudioRecording { + public enum RecorderError: Error, LocalizedError { + case converterUnavailable + + public var errorDescription: String? { + "Kon de microfoon niet omzetten naar 16 kHz mono." + } + } + + private let engine = AVAudioEngine() + private let lock = NSLock() + private var buffer: [Float] = [] + private let targetFormat = AVAudioFormat( + commonFormat: .pcmFormatFloat32, + sampleRate: 16_000, + channels: 1, + interleaved: false + )! + + public init() {} + + public func start() throws { + lock.withLock { buffer.removeAll(keepingCapacity: true) } + + let input = engine.inputNode + let inputFormat = input.outputFormat(forBus: 0) + guard let converter = AVAudioConverter(from: inputFormat, to: targetFormat) else { + throw RecorderError.converterUnavailable + } + let ratio = targetFormat.sampleRate / inputFormat.sampleRate + + input.installTap(onBus: 0, bufferSize: 4096, format: inputFormat) { [weak self] pcm, _ in + guard let self else { return } + let capacity = AVAudioFrameCount(Double(pcm.frameLength) * ratio) + 1 + guard let out = AVAudioPCMBuffer(pcmFormat: self.targetFormat, frameCapacity: capacity) else { return } + + var consumed = false + var error: NSError? + converter.convert(to: out, error: &error) { _, status in + if consumed { + status.pointee = .noDataNow + return nil + } + consumed = true + status.pointee = .haveData + return pcm + } + guard error == nil, let channel = out.floatChannelData?[0] else { return } + let samples = Array(UnsafeBufferPointer(start: channel, count: Int(out.frameLength))) + self.lock.withLock { self.buffer.append(contentsOf: samples) } + } + engine.prepare() + try engine.start() + } + + public func stop() -> [Float] { + engine.inputNode.removeTap(onBus: 0) + engine.stop() + return lock.withLock { + let samples = buffer + buffer = [] + return samples + } + } +} +``` + +`Sources/SstProbe/RecordProbe.swift`: + +```swift +import AVFoundation +import SstCore + +/// Records `seconds` of microphone audio; returns nil if access is denied or recording fails. +func recordSamples(seconds: Double) async -> [Float]? { + guard await AVCaptureDevice.requestAccess(for: .audio) else { + print("Microfoon-toegang geweigerd (Systeeminstellingen → Privacy → Microfoon).") + return nil + } + let recorder = AudioRecorder() + do { try recorder.start() } catch { + print("start mislukt:", error.localizedDescription) + return nil + } + print("Neem \(Int(seconds)) s op — praat nu") + try? await Task.sleep(for: .seconds(seconds)) + return recorder.stop() +} + +func runRecordProbe() async { + guard let samples = await recordSamples(seconds: 3) else { return } + let peak = samples.map { abs($0) }.max() ?? 0 + print("samples: \(samples.count) (verwacht ≈ 48000), piek: \(peak)") +} +``` + +`Sources/SstProbe/main.swift` (vervang volledig): + +```swift +import Foundation + +let command = CommandLine.arguments.dropFirst().first ?? "help" +switch command { +case "hotkey": + runHotkeyProbe() +case "record": + await runRecordProbe() +default: + print("usage: sst-probe hotkey|record") +} +``` + +- [ ] **Step 2: Bouwen** + +Run: `swift build` +Expected: build slaagt zonder errors. + +- [ ] **Step 3: Handmatige check** + +Run: `swift run SstProbe record` en praat 3 seconden. +Expected: `samples:` ligt rond 48000 (±2000) en `piek` is duidelijk > 0,05. Bij stilte is de piek ≈ 0. Bij een geweigerde microfoon staat er een duidelijke melding. + +- [ ] **Step 4: Commit** + +```bash +git add Sources +git commit -m "feat: add AudioRecorder (16 kHz mono) and record probe" -m "Co-Authored-By: Claude Sonnet 5.5 " +``` + +--- + +### Task 5: WhisperKitTranscriber en transcribe-probe + +**Files:** +- Create: `Sources/SstWhisper/WhisperKitTranscriber.swift` +- Create: `Sources/SstProbe/TranscribeProbe.swift` +- Modify: `Package.swift`, `Sources/SstProbe/main.swift` + +**Interfaces:** +- Consumes: `Transcriber` uit Task 1, `recordSamples(seconds:)` uit Task 4. +- Produces: + - `final class WhisperKitTranscriber: Transcriber` met `init(model: String = "large-v3-v20240930_turbo")`, `func load() async throws`, `WhisperKitTranscriber.TranscriberError: Error, LocalizedError` + - Probe-subcommando `transcribe` + +- [ ] **Step 1: Dependency en target toevoegen** + +`Package.swift` (vervang volledig): + +```swift +// swift-tools-version: 6.0 +import PackageDescription + +let package = Package( + name: "Sst", + platforms: [.macOS(.v14)], + dependencies: [ + .package(url: "https://github.com/argmaxinc/WhisperKit.git", from: "0.9.0"), + ], + targets: [ + .target(name: "SstCore"), + .target( + name: "SstWhisper", + dependencies: [ + "SstCore", + .product(name: "WhisperKit", package: "WhisperKit"), + ] + ), + .executableTarget(name: "SstProbe", dependencies: ["SstCore", "SstWhisper"]), + .testTarget(name: "SstCoreTests", dependencies: ["SstCore"]), + ], + swiftLanguageModes: [.v5] +) +``` + +- [ ] **Step 2: Implementatie** + +`Sources/SstWhisper/WhisperKitTranscriber.swift`: + +```swift +import Foundation +import SstCore +import WhisperKit + +public final class WhisperKitTranscriber: Transcriber { + public enum TranscriberError: Error, LocalizedError { + case notLoaded + + public var errorDescription: String? { "Spraakmodel is nog niet geladen." } + } + + private let modelName: String + private var pipeline: WhisperKit? + + public init(model: String = "large-v3-v20240930_turbo") { + self.modelName = model + } + + /// Downloads the model on first run (≈1.5 GB) and loads it into memory. + public func load() async throws { + pipeline = try await WhisperKit(WhisperKitConfig(model: modelName)) + } + + public func transcribe(_ samples: [Float]) async throws -> String { + guard let pipeline else { throw TranscriberError.notLoaded } + // language nil + detectLanguage: Whisper picks NL or EN per utterance. + let options = DecodingOptions( + task: .transcribe, + language: nil, + detectLanguage: true, + skipSpecialTokens: true, + withoutTimestamps: true + ) + let results = try await pipeline.transcribe(audioArray: samples, decodeOptions: options) + return results + .map(\.text) + .joined(separator: " ") + .trimmingCharacters(in: .whitespacesAndNewlines) + } +} +``` + +> Als de WhisperKit-API of het modelnaam-patroon in de opgeloste versie afwijkt (compilatiefout of "model not found"), pas dan alleen dit bestand aan. Beschikbare modelnamen: `WhisperKit.fetchAvailableModels()`. Houd `Transcriber` en `load()` gelijk, want Task 6 hangt daarvan af. + +`Sources/SstProbe/TranscribeProbe.swift`: + +```swift +import Foundation +import SstWhisper + +func runTranscribeProbe() async { + guard let samples = await recordSamples(seconds: 6) else { return } + print("Model laden (eerste keer: download ≈1.5 GB)…") + let transcriber = WhisperKitTranscriber() + do { + try await transcriber.load() + let started = Date.now + let text = try await transcriber.transcribe(samples) + let elapsed = Date.now.timeIntervalSince(started) + print(String(format: "(%.2f s) ", elapsed) + text) + } catch { + print("mislukt:", error.localizedDescription) + } +} +``` + +`Sources/SstProbe/main.swift` (vervang volledig): + +```swift +import Foundation + +let command = CommandLine.arguments.dropFirst().first ?? "help" +switch command { +case "hotkey": + runHotkeyProbe() +case "record": + await runRecordProbe() +case "transcribe": + await runTranscribeProbe() +default: + print("usage: sst-probe hotkey|record|transcribe") +} +``` + +- [ ] **Step 3: Bouwen** + +Run: `swift build` +Expected: dependencies worden opgehaald, build slaagt. Los API-afwijkingen op in `WhisperKitTranscriber.swift` (zie opmerking hierboven). + +- [ ] **Step 4: Handmatige check (NL, EN, gemengd)** + +Run: `swift run SstProbe transcribe` +Zeg achtereenvolgens: een Nederlandse zin, een Engelse zin, en een zin die begint in het Nederlands met een paar Engelse woorden. +Expected: correcte transcriptie van de zinnen; na het laden duurt transcriberen van 6 s audio ruim onder 3 s op de M4 Pro. Noteer de gemeten tijd. +Bekende beperking: taaldetectie is per opname, dus wisselen midden in een zin kan een van beide talen minder goed herkennen. Dat is acceptabel voor de skeleton. + +- [ ] **Step 5: Commit** + +```bash +git add Package.swift Package.resolved Sources +git commit -m "feat: add WhisperKit transcriber and transcribe probe" -m "Co-Authored-By: Claude Sonnet 5.5 " +``` + +--- + +### Task 6: Menubalk-app, app-bundle en rooktest + +**Files:** +- Create: `Sources/Sst/SstApp.swift`, `Sources/Sst/AppModel.swift`, `Sources/Sst/MenuContent.swift` +- Create: `scripts/make-app.sh` +- Modify: `Package.swift` + +**Interfaces:** +- Consumes: `DictationController`, `HotkeyMonitor`, `AudioRecorder`, `TextInserter`, `SystemPasteboard`, `CGEventPaster`, `PassthroughCleaner` (SstCore); `WhisperKitTranscriber` met `load()` (SstWhisper). +- Produces: de app-bundle `build/Sst.app`. + +- [ ] **Step 1: Executable target toevoegen** + +In `Package.swift`: voeg de regel `.executableTarget(name: "Sst", dependencies: ["SstCore", "SstWhisper"]),` toe aan `targets`, direct boven de `SstProbe`-regel. + +- [ ] **Step 2: App-code** + +`Sources/Sst/AppModel.swift`: + +```swift +import AVFoundation +import ApplicationServices +import SwiftUI +import SstCore +import SstWhisper + +@MainActor +final class AppModel: ObservableObject { + @Published var state: DictationState = .idle + @Published var modelStatus = "Model laden…" + @Published var modelReady = false + @Published var lastError: String? + @Published var micGranted = false + @Published var accessibilityGranted = false + @Published var inputMonitoringGranted = false + + private let transcriber = WhisperKitTranscriber() + private let hotkey = HotkeyMonitor() + private let controller: DictationController + + init() { + controller = DictationController( + recorder: AudioRecorder(), + transcriber: transcriber, + cleaner: PassthroughCleaner(), + inserter: TextInserter(pasteboard: SystemPasteboard(), paster: CGEventPaster()) + ) + controller.onStateChange = { [weak self] in self?.state = $0 } + controller.onError = { [weak self] in self?.lastError = $0 } + hotkey.onEvent = { [weak self] event in + guard let self, self.modelReady else { return } + Task { await self.controller.handle(event) } + } + Task { await start() } + } + + var statusLine: String { + guard modelReady else { return modelStatus } + switch state { + case .idle: return "Klaar — houd Fn ingedrukt om te dicteren" + case .recording: return "Luistert…" + case .transcribing: return "Transcriberen…" + case .inserting: return "Invoegen…" + } + } + + var iconName: String { + switch state { + case .idle: return modelReady ? "mic" : "mic.slash" + case .recording: return "mic.fill" + case .transcribing, .inserting: return "waveform" + } + } + + func refreshPermissions() { + micGranted = AVCaptureDevice.authorizationStatus(for: .audio) == .authorized + accessibilityGranted = AXIsProcessTrusted() + inputMonitoringGranted = CGPreflightListenEventAccess() + } + + func retryHotkey() { + refreshPermissions() + do { + try hotkey.start() + lastError = nil + } catch { + lastError = error.localizedDescription + } + } + + private func start() async { + _ = await AVCaptureDevice.requestAccess(for: .audio) + if !CGPreflightListenEventAccess() { _ = CGRequestListenEventAccess() } + if !AXIsProcessTrusted() { + _ = AXIsProcessTrustedWithOptions(["AXTrustedCheckOptionPrompt": true] as CFDictionary) + } + retryHotkey() + do { + try await transcriber.load() + modelReady = true + modelStatus = "Klaar" + } catch { + modelStatus = "Model laden mislukt" + lastError = error.localizedDescription + } + } +} +``` + +`Sources/Sst/MenuContent.swift`: + +```swift +import AppKit +import SwiftUI + +struct MenuContent: View { + @ObservedObject var model: AppModel + + var body: some View { + Text(model.statusLine) + if let error = model.lastError { + Text("Fout: \(error)") + } + Divider() + permissionRow("Microfoon", granted: model.micGranted, pane: "Privacy_Microphone") + permissionRow("Accessibility", granted: model.accessibilityGranted, pane: "Privacy_Accessibility") + permissionRow("Input Monitoring", granted: model.inputMonitoringGranted, pane: "Privacy_ListenEvent") + Button("Rechten opnieuw controleren") { model.retryHotkey() } + Divider() + Button("Stop") { NSApplication.shared.terminate(nil) } + .keyboardShortcut("q") + } + + @ViewBuilder + private func permissionRow(_ title: String, granted: Bool, pane: String) -> some View { + if granted { + Text("✓ \(title)") + } else { + Button("⚠︎ \(title) — open Instellingen") { + if let url = URL(string: "x-apple.systempreferences:com.apple.preference.security?\(pane)") { + NSWorkspace.shared.open(url) + } + } + } + } +} +``` + +`Sources/Sst/SstApp.swift`: + +```swift +import SwiftUI + +@main +struct SstApp: App { + @StateObject private var model = AppModel() + + var body: some Scene { + MenuBarExtra { + MenuContent(model: model) + } label: { + Image(systemName: model.iconName) + } + } +} +``` + +> Als de model-start pas begint na het openen van het menu (StateObject wordt lui aangemaakt), verplaats dan `AppModel` naar een `NSApplicationDelegateAdaptor`-delegate die het model in `applicationDidFinishLaunching` aanmaakt. + +- [ ] **Step 3: Bundle-script** + +`scripts/make-app.sh`: + +```bash +#!/usr/bin/env bash +set -euo pipefail +cd "$(dirname "$0")/.." + +swift build -c release --product Sst +BIN=".build/release/Sst" +APP="build/Sst.app" + +rm -rf "$APP" +mkdir -p "$APP/Contents/MacOS" "$APP/Contents/Resources" +cp "$BIN" "$APP/Contents/MacOS/Sst" +# SwiftPM resource bundles (if any dependency ships them) live next to the binary. +for b in .build/release/*.bundle; do + [ -e "$b" ] && cp -R "$b" "$APP/Contents/Resources/" +done + +cat > "$APP/Contents/Info.plist" <<'PLIST' + + + + + CFBundleIdentifierlocal.sst.app + CFBundleNameSst + CFBundleExecutableSst + CFBundlePackageTypeAPPL + CFBundleShortVersionString0.1.0 + CFBundleVersion1 + LSMinimumSystemVersion14.0 + LSUIElement + NSMicrophoneUsageDescription + Sst neemt je stem op zolang je Fn ingedrukt houdt, om die lokaal naar tekst om te zetten. + + +PLIST + +codesign --force --sign - --identifier local.sst.app "$APP" +echo "Gebouwd: $APP" +``` + +- [ ] **Step 4: Bouwen** + +Run: `chmod +x scripts/make-app.sh && ./scripts/make-app.sh` +Expected: `Gebouwd: build/Sst.app`, zonder errors. + +- [ ] **Step 5: Rooktest van de hele keten** + +Run: `open build/Sst.app` +Controleer in deze volgorde: +1. Mic-icoon verschijnt in de menubalk (geen Dock-icoon). Menu toont "Model laden…" en daarna "Klaar — houd Fn ingedrukt om te dicteren" (eerste keer duurt de download enkele minuten). +2. Geef Microfoon, Accessibility en Input Monitoring toe via de prompts of de knoppen in het menu; klik "Rechten opnieuw controleren" (zo nodig de app herstarten). Alle drie staan op ✓. +3. Open TextEdit, klik in een leeg document, houd Fn ingedrukt, zeg "Dit is een test van de dictatie", laat los. Icoon: `mic.fill` → `waveform` → `mic`. Tekst verschijnt op de cursor. +4. Herhaal in Notes en in een browser-tekstveld, met een Engelse en een gemengde zin. +5. Zet een eigen tekst op het klembord, dicteer, en controleer dat het klembord daarna weer je eigen tekst bevat. +6. Tik Fn kort aan (< 0,3 s): er mag niets ingevoegd worden. +7. Weiger Accessibility tijdelijk: het menu toont een foutmelding en de tekst staat op het klembord. + +Bekende beperking: omdat de app ad-hoc is gesigneerd (`codesign --sign -`), kan macOS de rechten na elke herbouw opnieuw vragen. Een vast ontwikkelcertificaat lost dat later op. + +- [ ] **Step 6: Commit** + +```bash +git add Package.swift Sources scripts +git commit -m "feat: add menu bar app and bundle script" -m "Co-Authored-By: Claude Sonnet 5.5 " +``` + +--- + +## Self-review + +- **Spec-dekking:** HotkeyMonitor (Task 3), AudioRecorder (4), Transcriber/WhisperKit (5), Cleaner-naad (2), TextInserter met klembordherstel en klembord-bij-falen (2), DictationController met statemachine, minimale opnameduur, lege transcriptie en foutpaden (1), MenuBarExtra met rechten-check en modelstatus (6). Niet gedekt, bewust: overlay-pill, cleanup, branding (latere specs). +- **Placeholders:** geen; alle codestappen bevatten volledige code. De twee opmerkingen over WhisperKit-API en StateObject zijn gerichte fallbacks, geen open eindjes. +- **Typeconsistentie:** `HotkeyEvent`, `DictationState`, `CleaningContext`, de vier protocollen en `DictationController.init(recorder:transcriber:cleaner:inserter:minimumSamples:)` zijn in Task 1 gedefinieerd en komen identiek terug in Task 2 t/m 6. `WhisperKitTranscriber.load()` uit Task 5 wordt in Task 6 gebruikt. +- **Risico's:** de WhisperKit-API en het exacte modelnaam-patroon zijn het enige onverifieerde punt; Task 5 stap 3 vangt dat vroeg af.