PHASE 7: Document Management & Data Import

Trigger: "Execute Phase 7 as per masterplan."

Goal: Document upload/storage with S3/MinIO, PDF generation for contracts and invoices, and a complete data import framework with CSV parser, mapping engine, dry-run capability, and pre-defined templates for 5 competitor systems.

Prerequisite: Phase 6 is completed.

Step 7.0 — Technical Conception

Create doc/developer/conception/phase-7-conception.md.

(reference: Chapter 11 — Data Model, Chapter 15 — Migration Strategy)

Content: - Scope: membership-document (upload, storage, PDF), membership-import (CSV, mapping, templates) - Storage Architecture: S3-compatible API (MinIO for local dev, Hetzner Object Storage for prod) - PDF Pipeline: Open HTML to PDF engine, Thymeleaf template rendering, per-entity branding - Import Framework: CSV parsing (Apache Commons CSV), column mapping engine, validation pipeline, dry-run mode - Template Design: 5 competitor templates with pre-configured field mappings - Error Handling: Per-row validation errors, downloadable error report, partial import with skip-on-error - Security: ClamAV virus scanning on upload, file type whitelist, max file size - Design Chapter References: Ch11, Ch15

Result: Technical conception for Phase 7.

---

Step 7.1 — Document Entity and Storage

Create the membership-document module.

(reference: Chapter 11 — Data Model)

Approach:

1. Create JPA entity: Document

Document fields:

Field	Type	Constraints	Description
id	Long	PK, generated
entityId	Long	FK NOT NULL	Organization
name	String(255)	NOT NULL	Original filename
displayName	String(255)		User-friendly display name
mimeType	String(100)	NOT NULL	MIME type (e.g., application/pdf)
sizeBytes	Long	NOT NULL	File size in bytes
storagePath	String(500)	NOT NULL	S3/MinIO path
linkedType	String(30)	NOT NULL	MEMBER / CONTRACT / ORGANIZATION / INVOICE / IMPORT
linkedId	Long	NOT NULL	ID of linked entity
category	String(30)	default 'OTHER'	CONTRACT_PDF / INVOICE_PDF / MEMBER_CARD / PHOTO / IMPORT_FILE / SEPA_XML / OTHER
generatedBy	String(30)	nullable	SYSTEM (auto-generated) / USER (uploaded)
uploadedBy	Long	FK nullable	User who uploaded (null if system-generated)
virusScanStatus	String(20)	default 'PENDING'	PENDING / CLEAN / INFECTED / SKIPPED
createdAt	Instant	auto
version	Long	@Version	Optimistic locking
2. Create StorageService (interface):
- upload(InputStream data, String path, String contentType) -> URL
- download(String path) -> InputStream
- delete(String path)
- getSignedUrl(String path, Duration expiry) -> String
3. Create MinioStorageService (S3-compatible implementation):
- Bucket: membership-documents
- Path convention: /{entityId}/{linkedType}/{linkedId}/{documentId}_{filename}
- Configurable: endpoint, access key, secret key, bucket name
4. Create DocumentService:
- Upload with metadata, download with access control
- File type whitelist: PDF, PNG, JPG, JPEG, DOCX, XLSX, CSV (configurable)
- Max file size: 10 MB (configurable)
- Virus scanning: ClamAV integration (optional, toggleable)
- Link document to member, contract, or organization
5. Create controller:
- POST /api/document/upload (multipart) — Upload document
- GET /api/document/{id} — Get document metadata
- GET /api/document/{id}/download — Download file
- DELETE /api/document/{id} — Delete document
- GET /api/document/list?linkedType=MEMBER&linkedId={id} — List documents for entity
6. Create Flyway migration: V700__create_document.sql
7. Add MinIO to backend/docker-compose.yml (port 9000, console 9001)
8. Create tests: DocumentServiceTest, MinioStorageServiceTest

Result: Document management with S3-compatible storage.

---

Step 7.2 — PDF Generation

(reference: Chapter 12 — Integration)

Approach:

1. Add openhtmltopdf-pdfbox dependency to membership-document
2. Create PdfGenerationService: - generateContractPdf(Contract contract) -> byte[] - generateInvoicePdf(Transaction transaction) -> byte[] - generateMemberCardPdf(Member member) -> byte[]
3. Create Thymeleaf templates in membership-document/src/main/resources/templates/pdf/: - contract-template.html — Contract PDF:
   • Header: organization logo (left), organization name + address (right)
   • Section 1: Member details (name, address, member number)
   • Section 2: Membership details (plan name, price, billing cycle, start date, minimum term)
   • Section 3: Terms and conditions (from CommunicationTemplate type AGB)
   • Section 4: Cancellation policy (notice period, auto-renewal clause)
   • Section 5: Signature area (digital signature data: name, date, IP address)
   • Footer: organization contact info, "Generated by Membership One"
   • invoice-template.html — Invoice PDF:
   • Header: organization logo, name, address, USt-IdNr.
   • Addressee: member name and address
   • Invoice details: invoice number, invoice date, due date, payment method
   • Line items table: position #, description, quantity, unit price, VAT rate, total
   • Summary: subtotal (net), VAT breakdown per rate (7%, 19%), total (gross)
   • Payment info: IBAN, BIC, bank name, mandate reference (if SEPA)
   • Footer: Amtsgericht, HRB, Geschäftsführer
   • member-card-template.html — Member card (A6 landscape):
   • Front: organization logo, member photo, full name, member number, QR code
   • Membership plan badge, valid until date
4. Per-entity branding: logo, primary color, accent color, footer text loaded from Organization settings JSONB
5. i18n: PDF rendered in member's preferred locale (from User.locale, falls back to entity locale)
6. Page format: A4 portrait for contracts/invoices, A6 landscape for member cards. Fonts: Noto Sans (includes RTL glyphs for Arabic/Hebrew)
7. Create controller: - GET /api/document/pdf/contract/{contractId} — Generate and download contract PDF - GET /api/document/pdf/invoice/{transactionId} — Generate and download invoice PDF
8. RabbitMQ integration: document.generate queue for async PDF generation of bulk documents
9. Create tests: PdfGenerationServiceTest

Result: PDF generation for contracts, invoices, member cards with branding and i18n.

---

Step 7.3 — Document Controller and Admin UI

1. Admin UI: document list per member/contract, upload drag-and-drop, preview (PDF/image inline)
2. Member app: view own documents, download contracts/invoices
3. Automatic document creation: contract PDF generated on purchase, invoice PDF on billing

Result: Document management integrated in admin and member UI.

---

Step 7.4 — Data Import Framework

Create the membership-import module.

(reference: Chapter 15 — Migration Strategy)

Approach:

1. Create JPA entities:

ImportJob fields:

Field	Type	Constraints	Description
id	Long	PK, generated
entityId	Long	FK NOT NULL	Organization
templateId	Long	FK NOT NULL	Import template used
fileName	String(255)	NOT NULL	Original CSV filename
fileStoragePath	String(500)	NOT NULL	S3/MinIO path to uploaded file
status	String(20)	default 'PENDING'	PENDING / VALIDATING / PREVIEWING / EXECUTING / COMPLETED / FAILED
totalRows	Integer	default 0	Total rows in CSV (excl. header)
validRows	Integer	default 0	Rows passing validation
importedRows	Integer	default 0	Successfully imported rows
errorRows	Integer	default 0	Rows with validation errors
errorReportPath	String(500)	nullable	S3 path to error CSV
startedAt	Instant	nullable	Processing start time
completedAt	Instant	nullable	Processing end time
createdBy	Long	FK NOT NULL	User who initiated
createdAt	Instant	auto
version	Long	@Version

ImportTemplate fields:

Field	Type	Constraints	Description
id	Long	PK, generated
name	String(200)	NOT NULL	Template display name (e.g., "easyVerein Members")
sourceSystem	String(50)	NOT NULL	EASY_VEREIN / CLUB_DESK / MAGICLINE / SPORTSCLUB / GENERIC
targetEntity	String(50)	NOT NULL	MEMBER / MEMBER_AND_CONTRACT / MEMBER_AND_BANK_ACCOUNT
columnMappings	JSONB	NOT NULL	[{"sourceColumn": "Vorname", "targetField": "firstName", "transform": null}, ...]
validationRules	JSONB	default '[]'	[{"field": "email", "rule": "EMAIL_FORMAT"}, {"field": "iban", "rule": "IBAN_CHECKSUM"}, ...]
dateFormat	String(30)	default 'yyyy-MM-dd'	Date parsing format (e.g., dd.MM.yyyy for German CSVs)
delimiter	String(1)	default ','	CSV delimiter (; common in DACH)
encoding	String(20)	default 'UTF-8'	File encoding
isSystemTemplate	boolean	default false	True = pre-defined (seed), false = user-created
createdAt	Instant	auto
version	Long	@Version
2. Create CsvParserService:
- Apache Commons CSV parser with configurable delimiter, quote char, encoding
- Header detection (first row), encoding auto-detection (UTF-8, ISO-8859-1, Windows-1252)
- Streaming: process file row-by-row (no full file in memory)
3. Create MappingEngine:
- Column mapping: source column -> target field (from ImportTemplate.columnMappings)
- Value transformations: date format conversion, status mapping, enum mapping
- Computed fields: full name = firstName + " " + lastName
4. Create ValidationPipeline:
- Required field checks
- Format validation: email, IBAN, date, phone (libphonenumber)
- Referential integrity: membership template exists, organization exists
- Duplicate detection: email, member number
- Per-row error collection (not fail-fast)
5. Create ImportExecutor:
- Dry-run mode: Validate all rows, report errors, create NO records
- Execute mode: After dry-run approval, create records in transaction batches (100 rows per transaction)
- Progress tracking via ImportJob.importedRows (updated every batch)
- Error report: CSV file with original row + error column
6. RabbitMQ: import.process queue for async execution
7. Create controller:
- POST /api/import/upload — Upload CSV file, create ImportJob
- POST /api/import/{jobId}/validate — Run dry-run validation
- GET /api/import/{jobId}/preview — Get validation results (errors, preview of first 10 rows)
- POST /api/import/{jobId}/execute — Execute import (after dry-run approval)
- GET /api/import/{jobId}/status — Get import progress
- GET /api/import/{jobId}/errors — Download error report CSV
- GET /api/import/templates — List available import templates
8. Create Flyway: V701__create_import.sql
9. Create tests: CsvParserServiceTest, MappingEngineTest, ValidationPipelineTest, ImportExecutorTest

Result: Complete import framework with CSV parsing, mapping, validation, and dry-run.

---

Step 7.5 — Pre-Defined Import Templates

(reference: Chapter 15 — Migration Strategy)

Approach:

Create 5 competitor import templates as seed data:

#	Source System	Target	Mapped Fields	Notes
1	easyVerein	Member + Contract	15 fields: name, email, address, phone, birthdate, IBAN, memberNumber, joinDate, membershipType, status, notes, emergencyContact, gender, newsletter, customFields	Most common competitor in DACH
2	ClubDesk	Member + Contract	12 fields: name, email, address, phone, memberType, entryDate, exitDate, bankAccount, memberNumber, notes, status, category	Popular in Switzerland
3	Magicline	Member + Contract + BankAccount	20 fields: fullName, email, phone, street, zip, city, country, dateOfBirth, gender, iban, bic, contractName, startDate, endDate, monthlyFee, paymentMethod, status, barcode, photo, notes	Fitness-specific (largest competitor)
4	SPORTSCLUB	Member + Contract	14 fields: firstName, lastName, email, mobile, address, birthDate, memberSince, membershipName, fee, paymentType, accountHolder, iban, bic, status	Sports club specific
5	Generic CSV	Member	User maps columns manually	Fallback for any system

Each template includes: - Column mapping configuration (JSONB) - Value transformation rules (status codes, date formats) - Sample CSV for testing

Result: 5 ready-to-use import templates covering 80%+ of market.

---

Step 7.6 — Admin Import UI

(reference: Chapter 08 — User Journeys)

Approach:

1. Import Wizard (4 steps): - Step 1 — Upload: Drag-and-drop CSV, select import template (or Generic), upload - Step 2 — Mapping: If Generic template: column mapping UI (source column dropdown -> target field). If pre-defined: show mapping summary, allow overrides - Step 3 — Preview: Show dry-run results: total rows, valid rows (green), error rows (red with details). Download error report. "Fix and re-upload" or "Import valid rows only" options - Step 4 — Execute: Progress bar (X of Y rows), status updates, completion summary

2. Import history: list of past import jobs with status, row counts, download links

Result: User-friendly import wizard with mapping, preview, and progress tracking.

---

Step 7.7 — Phase 7 Documentation

1. Update doc/enduser/admin-manual.md (document management, data import sections)
2. Update doc/enduser/consumer-manual.md (document access)
3. Update doc/developer/backend-guide.md (document and import module architecture)
4. Update doc/developer/api-reference.md (new endpoints)

---

Step 7.8 — Update Intranet

Run python doc/intranet/build.py.

---

Phase 7 — Quality Gate

#	Check	Target
1	Conception	exists
2	Compilation + tests	0 errors, 0 failures
3	Coverage	>= 70%
4	Document upload/download	works with access control
5	Contract PDF	generated with entity branding
6	Invoice PDF	includes VAT breakdown
7	CSV import dry-run	validates all 5 templates
8	CSV import execute	creates records correctly
9	Import error report	downloadable with row details
10	Documentation + intranet	updated
11	CLAUDE.md	updated

Report: "Phase 7 completed."

---