docs: add GitM design spec
Design document for Gitea repository sync tool with Go backend, Vue 3 frontend, SQLite storage, and single-binary deployment. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This commit is contained in:
panw
271
docs/superpowers/specs/2026-03-31-gitm-gitea-sync-design.md
Normal file
271
docs/superpowers/specs/2026-03-31-gitm-gitea-sync-design.md
Normal file
@@ -0,0 +1,271 @@
|
|||||||
|
# GitM - Gitea Repository Sync Tool Design
|
||||||
|
|
||||||
|
## Overview
|
||||||
|
|
||||||
|
GitM is a cross-platform tool that synchronizes all repositories from multiple Gitea servers to local storage. It runs as a single-binary web service with an embedded Vue 3 management UI, supporting both Windows and Linux.
|
||||||
|
|
||||||
|
## Tech Stack
|
||||||
|
|
||||||
|
- **Backend**: Go + Gin
|
||||||
|
- **Frontend**: Vue 3 + Element Plus + Pinia + Vite
|
||||||
|
- **Database**: SQLite (mattn/go-sqlite3)
|
||||||
|
- **Authentication**: JWT with simple password
|
||||||
|
- **Deployment**: Single binary with embedded frontend (Go embed)
|
||||||
|
|
||||||
|
## Architecture
|
||||||
|
|
||||||
|
```
|
||||||
|
┌─────────────────────────────────────────┐
|
||||||
|
│ gitm (single binary) │
|
||||||
|
│ │
|
||||||
|
│ ┌───────────────┐ ┌───────────────┐ │
|
||||||
|
│ │ Web UI │ │ API Server │ │
|
||||||
|
│ │ (Vue3+EP) │ │ (Gin) │ │
|
||||||
|
│ │ embedded │ │ │ │
|
||||||
|
│ └───────┬───────┘ └───────┬───────┘ │
|
||||||
|
│ │ │ │
|
||||||
|
│ ┌───────┴──────────────────┴───────┐ │
|
||||||
|
│ │ Business Logic │ │
|
||||||
|
│ │ - Server management (CRUD) │ │
|
||||||
|
│ │ - Repo discovery & sync │ │
|
||||||
|
│ │ - Scheduled task runner │ │
|
||||||
|
│ │ - Sync logging │ │
|
||||||
|
│ └───────────────┬───────────────┘ │
|
||||||
|
│ │ │
|
||||||
|
│ ┌───────────────┴───────────────┐ │
|
||||||
|
│ │ SQLite Database │ │
|
||||||
|
│ └───────────────────────────────┘ │
|
||||||
|
│ │
|
||||||
|
│ ┌───────────────────────────────┐ │
|
||||||
|
│ │ Local Storage │ │
|
||||||
|
│ │ repos/ │ │
|
||||||
|
│ │ ├── server1/ │ │
|
||||||
|
│ │ │ ├── owner1/ │ │
|
||||||
|
│ │ │ │ └── repo1.git/ │ │
|
||||||
|
│ │ │ └── owner2/ │ │
|
||||||
|
│ │ └── server2/ │ │
|
||||||
|
│ └───────────────────────────────┘ │
|
||||||
|
└───────────────────────────────────────┘
|
||||||
|
```
|
||||||
|
|
||||||
|
## Data Model
|
||||||
|
|
||||||
|
### gitea_servers
|
||||||
|
|
||||||
|
| Field | Type | Description |
|
||||||
|
|-------|------|-------------|
|
||||||
|
| id | INTEGER PK | Auto-increment ID |
|
||||||
|
| name | TEXT NOT NULL | Display name / alias |
|
||||||
|
| url | TEXT NOT NULL | Gitea server URL (e.g. `https://git.example.com`) |
|
||||||
|
| token | TEXT NOT NULL | API access token |
|
||||||
|
| sync_interval | INTEGER DEFAULT 0 | Sync interval in minutes (0 = manual only) |
|
||||||
|
| last_sync_at | DATETIME | Last successful sync time |
|
||||||
|
| status | TEXT DEFAULT 'active' | active / disabled |
|
||||||
|
| created_at | DATETIME | Creation time |
|
||||||
|
|
||||||
|
### repos
|
||||||
|
|
||||||
|
| Field | Type | Description |
|
||||||
|
|-------|------|-------------|
|
||||||
|
| id | INTEGER PK | Auto-increment ID |
|
||||||
|
| server_id | INTEGER FK | References gitea_servers.id |
|
||||||
|
| name | TEXT | Repository name |
|
||||||
|
| full_name | TEXT | Full path (owner/repo) |
|
||||||
|
| clone_url | TEXT | Clone URL |
|
||||||
|
| local_path | TEXT | Local storage path |
|
||||||
|
| size | INTEGER | Repository size in bytes |
|
||||||
|
| last_sync_at | DATETIME | Last sync time |
|
||||||
|
| sync_status | TEXT | syncing / success / failed / pending |
|
||||||
|
| created_at | DATETIME | Discovery time |
|
||||||
|
|
||||||
|
### sync_logs
|
||||||
|
|
||||||
|
| Field | Type | Description |
|
||||||
|
|-------|------|-------------|
|
||||||
|
| id | INTEGER PK | Auto-increment ID |
|
||||||
|
| server_id | INTEGER FK | References gitea_servers.id |
|
||||||
|
| repo_id | INTEGER FK | References repos.id (nullable for full sync) |
|
||||||
|
| status | TEXT | success / failed / in_progress |
|
||||||
|
| message | TEXT | Log message or error details |
|
||||||
|
| started_at | DATETIME | Start time |
|
||||||
|
| finished_at | DATETIME | End time |
|
||||||
|
|
||||||
|
### settings
|
||||||
|
|
||||||
|
| Field | Type | Description |
|
||||||
|
|-------|------|-------------|
|
||||||
|
| key | TEXT PK | Setting key |
|
||||||
|
| value | TEXT | Setting value |
|
||||||
|
|
||||||
|
Predefined settings: `admin_password`, `listen_addr` (default `:9000`), `repos_dir`, `max_concurrent` (default 3).
|
||||||
|
|
||||||
|
## Gitea API Integration
|
||||||
|
|
||||||
|
### Authentication
|
||||||
|
|
||||||
|
All API requests use `Authorization: token <token>` header or `token=<token>` query parameter.
|
||||||
|
|
||||||
|
### Core API Endpoints Used
|
||||||
|
|
||||||
|
1. **Validate token**: `GET /api/v1/user` - verify token and get user info
|
||||||
|
2. **Search repos**: `GET /api/v1/repos/search?limit=50&page=1` - paginated list of all accessible repos
|
||||||
|
3. **Admin repos** (if admin token): `GET /api/v1/admin/repos` - all repos on server
|
||||||
|
|
||||||
|
### Sync Flow
|
||||||
|
|
||||||
|
```
|
||||||
|
For each active server (per schedule or manual trigger):
|
||||||
|
|
||||||
|
1. Validate token → GET /api/v1/user
|
||||||
|
2. Discover repos → GET /api/v1/repos/search (paginate through all pages)
|
||||||
|
3. Compare with database:
|
||||||
|
- New repos → git clone --mirror <url> to repos/<server>/<owner>/<repo>.git
|
||||||
|
- Existing repos → git fetch --all --prune (incremental update)
|
||||||
|
- Repos removed from Gitea → mark as deleted in DB (optional cleanup)
|
||||||
|
4. Record sync result to sync_logs
|
||||||
|
5. Update repos table (last_sync_at, sync_status)
|
||||||
|
```
|
||||||
|
|
||||||
|
### Concurrency Control
|
||||||
|
|
||||||
|
- Worker pool with configurable max concurrency (default: 3)
|
||||||
|
- Each server sync is serialized (repos within a server sync concurrently up to limit)
|
||||||
|
- Long-running sync operations return task_id for status polling
|
||||||
|
|
||||||
|
## API Routes
|
||||||
|
|
||||||
|
### Authentication
|
||||||
|
|
||||||
|
```
|
||||||
|
POST /api/login # Password auth, returns JWT token
|
||||||
|
```
|
||||||
|
|
||||||
|
### Settings
|
||||||
|
|
||||||
|
```
|
||||||
|
GET /api/settings # Get all settings (password masked)
|
||||||
|
PUT /api/settings # Update settings
|
||||||
|
```
|
||||||
|
|
||||||
|
### Server Management
|
||||||
|
|
||||||
|
```
|
||||||
|
GET /api/servers # List all servers
|
||||||
|
POST /api/servers # Add server
|
||||||
|
PUT /api/servers/:id # Update server
|
||||||
|
DELETE /api/servers/:id # Delete server
|
||||||
|
POST /api/servers/:id/test # Test connection (validate token)
|
||||||
|
```
|
||||||
|
|
||||||
|
### Repository Management
|
||||||
|
|
||||||
|
```
|
||||||
|
GET /api/servers/:id/repos # List repos for a server
|
||||||
|
POST /api/servers/:id/discover # Trigger repo discovery from Gitea API
|
||||||
|
```
|
||||||
|
|
||||||
|
### Sync Operations
|
||||||
|
|
||||||
|
```
|
||||||
|
POST /api/servers/:id/sync # Trigger sync for one server
|
||||||
|
POST /api/sync/all # Trigger sync for all servers
|
||||||
|
GET /api/servers/:id/sync/status # Get current sync status
|
||||||
|
```
|
||||||
|
|
||||||
|
### Logs & Stats
|
||||||
|
|
||||||
|
```
|
||||||
|
GET /api/sync/logs # Sync logs (paginated, filterable)
|
||||||
|
GET /api/sync/stats # Dashboard stats
|
||||||
|
```
|
||||||
|
|
||||||
|
All routes except `/api/login` require `Authorization: Bearer <token>` header.
|
||||||
|
|
||||||
|
## Web UI Pages
|
||||||
|
|
||||||
|
1. **Dashboard** - Overview: server count, total repos, disk usage, recent sync status
|
||||||
|
2. **Server Management** - Add/edit/delete Gitea servers, test connection, trigger sync
|
||||||
|
3. **Repository List** - Filter by server, show name, size, sync status, last sync time
|
||||||
|
4. **Sync Logs** - Sync history, filter by status (success/failed)
|
||||||
|
5. **Settings** - Change password, listen address, storage path, concurrency config
|
||||||
|
|
||||||
|
## Project Structure
|
||||||
|
|
||||||
|
```
|
||||||
|
gitm/
|
||||||
|
├── main.go # Entry point: parse flags, start server
|
||||||
|
├── go.mod
|
||||||
|
├── go.sum
|
||||||
|
├── internal/
|
||||||
|
│ ├── config/ # Configuration management
|
||||||
|
│ ├── database/ # SQLite init & migration
|
||||||
|
│ ├── models/ # Data model definitions
|
||||||
|
│ ├── gitea/ # Gitea API client
|
||||||
|
│ ├── sync/ # Sync engine (clone/fetch/scheduler)
|
||||||
|
│ ├── middleware/ # JWT auth middleware
|
||||||
|
│ └── handler/ # HTTP handlers (API route handlers)
|
||||||
|
├── web/ # Vue 3 frontend source
|
||||||
|
│ ├── src/
|
||||||
|
│ │ ├── views/ # Page components
|
||||||
|
│ │ ├── components/ # Shared components
|
||||||
|
│ │ ├── api/ # API call wrappers
|
||||||
|
│ │ └── stores/ # Pinia state management
|
||||||
|
│ ├── dist/ # Build output (embed target)
|
||||||
|
│ └── vite.config.ts
|
||||||
|
├── Makefile # Build & package commands
|
||||||
|
└── docs/ # Documentation
|
||||||
|
```
|
||||||
|
|
||||||
|
## Build & Run
|
||||||
|
|
||||||
|
### Build
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Build frontend
|
||||||
|
cd web && npm install && npm run build
|
||||||
|
|
||||||
|
# Build binary
|
||||||
|
go build -o gitm .
|
||||||
|
|
||||||
|
# Cross-compile
|
||||||
|
GOOS=linux GOARCH=amd64 go build -o gitm-linux .
|
||||||
|
GOOS=windows GOARCH=amd64 go build -o gitm.exe .
|
||||||
|
```
|
||||||
|
|
||||||
|
### Run
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# First run (init database, set password)
|
||||||
|
./gitm --init
|
||||||
|
|
||||||
|
# Normal run (default port 9000)
|
||||||
|
./gitm
|
||||||
|
|
||||||
|
# Custom port
|
||||||
|
./gitm --addr :9090
|
||||||
|
|
||||||
|
# Custom data directory
|
||||||
|
./gitm --data /path/to/data
|
||||||
|
```
|
||||||
|
|
||||||
|
### Runtime Flags
|
||||||
|
|
||||||
|
| Flag | Default | Description |
|
||||||
|
|------|---------|-------------|
|
||||||
|
| `--addr` | `:9000` | Listen address |
|
||||||
|
| `--data` | `./data` | Data directory (SQLite DB + repos) |
|
||||||
|
| `--init` | false | Initialize config and set password |
|
||||||
|
|
||||||
|
## Error Handling
|
||||||
|
|
||||||
|
- Gitea API errors: log and continue with next repo, mark failed in sync_logs
|
||||||
|
- Git clone/fetch failures: retry once, then mark as failed
|
||||||
|
- Network timeouts: 30s for API calls, no timeout for git operations (large repos)
|
||||||
|
- Disk space: check available space before sync, warn if below 1GB
|
||||||
|
|
||||||
|
## Security
|
||||||
|
|
||||||
|
- Password hashed with bcrypt before storage
|
||||||
|
- JWT tokens with configurable expiry (default 24h)
|
||||||
|
- Gitea tokens encrypted at rest using AES-256 with a key derived from the admin password
|
||||||
|
- API not accessible externally by default (bind to localhost, configurable)
|
||||||
Reference in New Issue
Block a user