Backup & Recovery Procedures

Comprehensive backup and disaster recovery procedures for MDHosting Ltd infrastructure, ensuring business continuity and data protection.

Overview

MDHosting's backup strategy is designed to protect client data and ensure rapid recovery in the event of data loss, hardware failure, or security incidents. All backups are stored off-site at Hetzner's dedicated backup storage facility for maximum resilience.

Backup Philosophy: - Defence in Depth: Multiple backup retention periods (daily, weekly, monthly) - Off-Site Storage: Backups stored separately from production servers - Automated Operations: Daily automated backups with error notification - GDPR Compliance: Client data backups follow data retention policies - Tested Recovery: Regular testing ensures backups are functional

Key Metrics: - RPO (Recovery Point Objective): 24 hours (daily backups) - RTO (Recovery Time Objective): <4 hours for critical systems, <24 hours for full server - Backup Success Rate Target: 99%+ - Off-Site Storage: Hetzner BX11 Storage Box (separate infrastructure)

Current Backup Infrastructure

Backup Architecture

graph TB
    subgraph "Production Servers - Hetzner Germany"
        EU1[eu1.cp<br/>CPX31 Hosting Server<br/>~30 Client Accounts]
        NS1[ns1.mdhosting.co.uk<br/>DNS Server 1]
        NS2[ns2.mdhosting.co.uk<br/>DNS Server 2]
    end

    subgraph "Backup Storage - Hetzner Germany"
        BX11[Hetzner BX11<br/>Storage Box<br/>Off-Site Backup Storage]
    end

    subgraph "Backup Schedule"
        DAILY[Daily Backups<br/>Automated via cPanel]
        WEEKLY[Weekly Retention]
        MONTHLY[Monthly Retention]
    end

    EU1 -->|SSH/SFTP Transfer| BX11
    NS1 -->|Configuration Backups| BX11
    NS2 -->|Configuration Backups| BX11

    DAILY --> WEEKLY
    WEEKLY --> MONTHLY

    BX11 -->|Recovery Operations| EU1

    classDef production fill:#3498db,stroke:#2c3e50,stroke-width:2px,color:#fff
    classDef backup fill:#27ae60,stroke:#2c3e50,stroke-width:2px,color:#fff
    classDef schedule fill:#f39c12,stroke:#2c3e50,stroke-width:2px,color:#fff

    class EU1,NS1,NS2 production
    class BX11 backup
    class DAILY,WEEKLY,MONTHLY schedule

Backup Solution

Primary Backup System: cPanel/WHM Backup Configuration

Technology Stack: - Backup Engine: cPanel Backup System (built-in) - Transfer Method: SSH/SFTP to remote storage - Storage Location: Hetzner BX11 Storage Box (off-site) - Scheduling: cron-based daily execution - Notification: Email alerts on backup failures

Storage Box Specifications: - Provider: Hetzner Storage Box BX11 - Location: Hetzner Germany datacenter (separate from hosting) - Capacity: 1 TB - Access: SSH/SFTP, CIFS/Samba, FTP - Redundancy: Hetzner-managed RAID storage

What Gets Backed Up

eu1.cp (Primary Hosting Server)

User Account Data: - ✅ Home directories (/home/username/) - Website files (public_html, subdomains) - Email data (mail directories) - Private files (.ssh, logs, temporary files) - Configuration files (.htaccess, .user.ini)

Databases: - ✅ MySQL/MariaDB databases - All client databases - Database users and permissions - Full database dumps in SQL format

Email Accounts: - ✅ Mailboxes (IMAP/POP3 mail storage) - ✅ Email filters and forwarding rules - ✅ Autoresponders configuration - ✅ Mailing lists

DNS Zones: - ✅ All DNS zone files - ✅ Zone configuration and serial numbers - ✅ Custom DNS records

Server Configuration: - ✅ cPanel account configurations - ✅ Apache/nginx virtual host configurations - ✅ PHP-FPM pool configurations - ✅ SSL/TLS certificates (Let's Encrypt) - ✅ Email server configuration (Exim, Dovecot) - ✅ FTP account configurations - ✅ Cron jobs

NOT Backed Up (By Design): - ❌ Operating system files (reinstall from ISO) - ❌ cPanel/WHM software (reinstall from repo) - ❌ System packages (yum/dnf reinstall) - ❌ Logs older than retention period - ❌ Temporary files and caches

DNS Servers (ns1, ns2)

DNS Configuration: - ✅ cPanel DNS zone files - ✅ PowerDNS configuration (when migrated) - ✅ Server configuration files

Note: DNS zones are primarily backed up from eu1.cp as the DNS master. DNS servers act as secondaries.

Backup Schedule & Retention

Backup Timing

Daily Backups: - Frequency: Every 24 hours - Execution Time: Automated via cron (exact time configured in cPanel) - Type: Full backups (entire account data) - Duration: 1-3 hours depending on data volume

Backup Window

Backups run during off-peak hours to minimise performance impact. Check WHM > Backup Configuration for exact schedule.

Retention Policy

Standard Retention Schedule:

Retention Period	Number of Copies	Description
Daily	7 days	Last 7 daily backups
Weekly	4 weeks	One backup per week for 4 weeks
Monthly	3 months	One backup per month for 3 months

Retention Example:

Daily:   [Day 1] [Day 2] [Day 3] [Day 4] [Day 5] [Day 6] [Day 7]
Weekly:  [Week 1] [Week 2] [Week 3] [Week 4]
Monthly: [Month 1] [Month 2] [Month 3]

Total Storage: Approximately 14 backup snapshots at any time (7 daily + 4 weekly + 3 monthly)

Automatic Rotation: - Oldest daily backup deleted after 7 days - Oldest weekly backup deleted after 4 weeks - Oldest monthly backup deleted after 3 months - Rotation handled automatically by cPanel backup system

Storage Management

Local Storage (Production Server): - ⚠️ Backups are removed from production server after transfer to off-site storage - Minimises disk usage on hosting server (160GB NVMe) - Reduces risk of simultaneous server and backup loss

Remote Storage (Hetzner BX11): - ✅ All backups stored on Hetzner Storage Box - 1 TB capacity for backup retention - Separate physical infrastructure from hosting servers - Accessible via SSH/SFTP for recovery operations

Current Usage Monitoring:

# Check storage box usage (from local machine with SSH access)
ssh [storage-box-user]@[storage-box-host] df -h

# List backups
ssh [storage-box-user]@[storage-box-host] ls -lh /path/to/backups/

Backup Verification

Automated Monitoring

cPanel Backup Notifications: - ✅ Email alerts sent to admin@mdhosting.co.uk - ✅ Success notifications (daily summary) - ✅ Failure notifications (immediate alert) - ✅ Backup log files in WHM

Email Alert Examples:

Success Notification:

Subject: Backup Success - eu1.cp - [Date]

Daily backup completed successfully.
- Accounts backed up: 30
- Databases backed up: 75
- Total size: 45 GB
- Duration: 2 hours 15 minutes
- Destination: Hetzner BX11 Storage Box

Failure Notification:

Subject: ⚠️ BACKUP FAILURE - eu1.cp - [Date]

Backup failed for the following accounts:
- clientdomain.com: Database dump failed (connection timeout)

Action Required: Investigate and retry manually.

Manual Verification

Monthly Backup Audit (Recommended):

Access Storage Box:

sftp [storage-box-user]@[storage-box-host]

Verify Recent Backups Exist:

ls -lh /backups/daily/
ls -lh /backups/weekly/
ls -lh /backups/monthly/

Check Backup Sizes:
Compare recent backup sizes for anomalies
Sudden size changes may indicate issues
Document baseline sizes for comparison
Test Backup Integrity:
Download a small account backup
Extract and verify archive integrity
Check database dumps are readable

Backup Integrity Test Procedure:

# Download a small backup
scp [storage-box]:/backups/daily/[account]-[date].tar.gz /tmp/

# Extract and verify
cd /tmp
tar -tzf [account]-[date].tar.gz | head -20

# Check database dump
tar -xzf [account]-[date].tar.gz mysql/[database].sql
head -50 mysql/[database].sql

Recovery Procedures

Recovery Scenarios

MDHosting recovery procedures cover four primary scenarios:

File Recovery - Restore individual files or directories
Account Recovery - Restore complete client account
Server Recovery - Rebuild failed server from backups
Disaster Recovery - Complete infrastructure failure

1. File Recovery (Individual Files/Folders)

Use Case: Client accidentally deleted files, restore specific files from backup.

Recovery Time: 15-30 minutes

Procedure:

Identify Required Backup:
Determine when file was last known to exist
Select appropriate backup date (daily/weekly/monthly)

Access Backup via WHM:

WHM → Backup → Backup Restoration
→ Select Account
→ Select Backup Date
→ Browse Files

Download Specific Files:
Navigate to required directory in backup
Download files to local machine
Verify file integrity
Restore to Client Account:
Upload files via cPanel File Manager or SFTP
Set correct ownership and permissions
Verify file functionality
Notify Client:
Confirm files restored successfully
Document restoration in support ticket

Alternative Method (Command Line):

# SSH to server
ssh root@eu1.cp

# Download backup from storage box
scp [storage-box]:/backups/daily/[account]-[date].tar.gz /tmp/

# Extract specific files
cd /tmp
tar -xzf [account]-[date].tar.gz home/[user]/public_html/[specific-file]

# Copy to client account
cp -a home/[user]/public_html/[specific-file] /home/[user]/public_html/

# Fix ownership
chown [user]:[user] /home/[user]/public_html/[specific-file]

# Clean up
rm -rf /tmp/[account]-[date].tar.gz /tmp/home

2. Database Recovery

Use Case: Database corruption, accidental table drop, restore to previous state.

Recovery Time: 15-45 minutes

Procedure:

Access Backup:

WHM → Backup → Backup Restoration
→ Select Account
→ Select Backup Date
→ MySQL Databases section

Download Database Dump:
Download .sql file for required database
Verify SQL dump is readable (check with text editor)
Prepare for Restoration:
CRITICAL: Backup current database state first (if any data exists)
Drop existing database (if replacing completely)
Or restore to new database name for comparison

Restore Database:

# Via command line
mysql -u root -p
CREATE DATABASE [database_name];
GRANT ALL ON [database_name].* TO '[user]'@'localhost';
exit

mysql -u root -p [database_name] < /tmp/database_backup.sql

Or via cPanel:

cPanel → phpMyAdmin
→ Select Database
→ Import tab
→ Choose SQL file
→ Execute

Verify Restoration:
Check table count and record counts
Test application functionality
Verify client website/application works
Clean Up:
Remove temporary SQL dump files
Document restoration

3. Full Account Recovery

Use Case: Complete account restoration after data loss, migration, or client request.

Recovery Time: 1-4 hours (depending on account size)

Procedure:

Preparation:
Identify backup date to restore from
Verify account doesn't currently exist (or will be overwritten)
Notify client of maintenance window if live

Download Backup:

# SSH to server
ssh root@eu1.cp

# Download backup from storage box
scp [storage-box]:/backups/[period]/cpmove-[account].tar.gz /home/

Restore via WHM:

WHM → Transfer Tool → Restore a Full Backup/cpmove File
→ Upload backup file (or select from /home/)
→ Click "Restore"
→ Wait for completion (monitor progress)

Verify Restoration:
✅ Website files present and accessible
✅ Databases restored and functional
✅ Email accounts operational (send/receive test)
✅ DNS records correct
✅ SSL certificate active
✅ Cron jobs configured
✅ Subdomain/addon domains working

Post-Restoration Checks:

# Check account exists
cat /etc/userdomains | grep [domain]

# Check disk usage
du -sh /home/[username]

# Check databases
mysql -e "SHOW DATABASES LIKE '[username]_%';"

# Check email
ls -lh /home/[username]/mail/[domain]/

# Test website
curl -I https://[domain].com

Client Notification:
Confirm restoration complete
Request client verify functionality
Document restoration details

4. Full Server Recovery (Disaster Recovery)

Use Case: Complete server failure, hardware replacement, catastrophic data loss.

Recovery Time: 4-24 hours (depending on data volume)

RTO Target: 24 hours RPO: 24 hours (last daily backup)

Prerequisites: - Access to Hetzner Storage Box with backups - New server provisioned (or existing server reinstalled) - Root SSH access to new server - cPanel/WHM license transferred or reactivated

Procedure:

Phase 1: Server Preparation (1-2 hours)

Provision New Server:
Deploy new Hetzner CPX31 (or equivalent)
Install AlmaLinux 8 (or AlmaLinux 10 for ApisCP migration)
Configure network and hostname
Update system packages

Install cPanel/WHM:

# Download and install cPanel
cd /home
curl -o latest -L https://securedownloads.cpanel.net/latest
sh latest
# Installation takes 30-60 minutes

Basic Server Configuration:
Configure hostname
Set up CSF firewall rules
Configure SSH (non-standard port, key auth)
Install additional required software
Configure Backup Destination:
Reconnect to Hetzner Storage Box
Configure SSH keys for backup access
Test connectivity to storage box

Phase 2: Backup Restoration (2-8 hours)

Download Backups from Storage Box:

# Create temporary directory
mkdir -p /backup_restore
cd /backup_restore

# Download all account backups from most recent daily backup
scp [storage-box]:/backups/daily/cpmove-*.tar.gz ./

# Or download specific accounts
scp [storage-box]:/backups/daily/cpmove-[account].tar.gz ./

Restore Accounts:

# Method 1: Via WHM (recommended for small batches)
WHM → Transfer Tool → Restore a Full Backup/cpmove File
→ Select each backup file
→ Restore one by one

# Method 2: Command line batch restore (faster for many accounts)
cd /backup_restore
for backup in cpmove-*.tar.gz; do
    /scripts/restorepkg $backup
    echo "Restored: $backup"
done

Monitor Restoration Progress:
Check /usr/local/cpanel/logs/cpbackup/ for logs
Monitor server load and disk space
Verify each account after restoration

Phase 3: Service Verification (1-2 hours)

DNS Configuration:
Update DNS records if server IP changed
Configure nameservers (ns1, ns2)
Wait for DNS propagation (up to 48 hours)
Use hosts file overrides for immediate testing

Verify All Services:

# Check web services
systemctl status httpd
systemctl status nginx

# Check email services
systemctl status exim
systemctl status dovecot

# Check databases
systemctl status mysql
mysql -e "SHOW DATABASES;"

# Check cPanel services
/scripts/restartsrv_httpd
/scripts/restartsrv_cpanel

Account-by-Account Verification:
- Test website access for each domain
- Send test email (SMTP)
- Receive test email (IMAP/POP3)
- Verify database connectivity
- Check SSL certificates (reissue if needed)
- Test cPanel access for sample accounts

Phase 4: Post-Recovery (1-2 hours)

Security Hardening:
- Reconfigure CSF firewall
- Enable Fail2Ban
- Configure SSH restrictions
- Review and lock down unnecessary services
- Run ClamAV malware scan
Monitoring Setup:
- Configure backup system to Storage Box
- Run initial backup to verify
- Set up monitoring alerts
- Configure Wazuh agent (if deployed)
Client Communication:
- Notify all clients of recovery completion
- Request clients verify their services
- Provide new IP address if changed
- Update support documentation
Documentation:
- Document cause of failure
- Record recovery timeline
- Note any issues encountered
- Update disaster recovery procedures
- Calculate RTO/RPO achieved

Disaster Recovery Checklist

Client Self-Service Restore

cPanel Backup Interface

Clients can restore their own files and databases without administrator intervention:

Access Method:

cPanel → Files → Backup Wizard

Available Restore Options:

Home Directory Restore:
Download full home directory backup
Restore via File Manager or FTP
Self-service file recovery
Database Restore:
Download database backups (.sql.gz files)
Restore via phpMyAdmin
Self-service database recovery
Email Forwarders and Filters:
Included in home directory backup
Restore via Email section

Limitations: - Only most recent backup available (if enabled) - May not include all retention periods - Administrator has access to full retention (daily/weekly/monthly)

Client Guidance:

Client Self-Service Restore Guide

To restore your own files: 1. Log in to cPanel 2. Navigate to Files → Backup Wizard 3. Click "Restore" 4. Select "Home Directory" or "MySQL Database" 5. Download the backup file 6. Extract and upload files as needed

If you need backups older than the most recent backup, contact support.

Backup Configuration

cPanel/WHM Backup Settings

Access Backup Configuration:

WHM → Backup → Backup Configuration

Current Configuration Settings:

Backup Type: - ✅ Full backups (complete account data) - Frequency: Daily - Retention: See retention policy above

Backup Destinations: - ✅ Remote destination: Hetzner BX11 Storage Box - Transfer type: SSH/SFTP - Compression: gzip (tar.gz format) - ❌ Local retention: Disabled (backups removed after upload)

Backup Scope: - ✅ User accounts (home directories) - ✅ Databases (MySQL dumps) - ✅ DNS zones - ✅ Server configuration - ✅ Email accounts and data

Notification Settings: - ✅ Email on backup completion: admin@mdhosting.co.uk - ✅ Email on backup failure: admin@mdhosting.co.uk - Log location: /usr/local/cpanel/logs/cpbackup/

Storage Box Configuration

Hetzner Storage Box Details: - Product: BX11 (1 TB capacity) - Access: SSH/SFTP (primary), FTP, CIFS/Samba (available) - Location: Hetzner datacentre (separate from hosting servers) - Authentication: SSH key (recommended) + password backup

Access Configuration:

# Test storage box connectivity
ssh -p 23 [storage-box-user]@[storage-box-host]

# SFTP access
sftp -P 23 [storage-box-user]@[storage-box-host]

# Mount as filesystem (optional, for manual operations)
sshfs [storage-box-user]@[storage-box-host]:/ /mnt/backups -p 23

Directory Structure:

/backups/
├── daily/
│   ├── cpmove-account1-[date].tar.gz
│   ├── cpmove-account2-[date].tar.gz
│   └── ...
├── weekly/
│   └── [weekly backups]
└── monthly/
    └── [monthly backups]

Backup Testing Schedule

Regular backup testing is essential to ensure recovery capabilities. MDHosting follows a structured testing schedule:

Monthly Testing

When: First Monday of each month Duration: 30-45 minutes Responsibility: System Administrator

Test Procedure:

Verify Backup Completion:
Check email notifications for last 7 days
Review backup logs in WHM
Confirm no failed backups
Storage Box Audit:
SSH to storage box
List all backup files
Check storage capacity usage
Verify retention policy is working
Test File Restore:
Select one client account at random
Download latest backup
Extract and verify 5-10 random files
Verify file integrity
Document Results:
Record test date and outcomes
Note any issues discovered
Update this documentation if needed

Quarterly Testing

When: First Monday of January, April, July, October Duration: 1-2 hours Responsibility: System Administrator

Test Procedure:

All Monthly Tests (as above)
Database Restore Test:
Select one database from backup
Restore to test database name
Verify table structure and record counts
Delete test database
Full Account Restore Test:
Select small client account (or create test account)
Perform full account restoration to test account name
Verify all components (files, databases, email, DNS)
Delete test account after verification
Document Results:
Calculate restoration time (actual RTO)
Record any issues or delays
Update disaster recovery estimates

Annual Testing

When: January (planning year) Duration: 4-8 hours Responsibility: System Administrator + Team Review

Test Procedure:

All Monthly and Quarterly Tests (as above)
Simulated Disaster Recovery:
Provision test server
Attempt full server recovery from backups
Time the entire process
Document every step
Backup Strategy Review:
Review retention policy adequacy
Assess storage capacity requirements
Evaluate backup technology (alternatives to cPanel backup)
Consider additional off-site locations
Update Documentation:
Update disaster recovery procedures based on simulation
Revise RTO/RPO targets if needed
Document lessons learned

Annual Report: - Backup success rate for the year - Number of restores performed - Average restoration time - Storage usage trends - Recommendations for improvements

Troubleshooting

Common Backup Issues

1. Backup Fails with "Disk Space Full"

Symptoms: - Backup error: "No space left on device" - Backup incomplete or failed

Diagnosis:

# Check server disk usage
df -h

# Check storage box usage
ssh [storage-box] df -h

# Check largest directories
du -h /home/ | sort -rh | head -20

Resolution: - If server disk full: Clear temporary files, old logs - If storage box full: Adjust retention policy or upgrade storage - Remove old backups manually if needed - Consider excluding log directories from backups

2. Backup Transfer Fails to Storage Box

Symptoms: - Backup completes locally but doesn't transfer - Email notification: "Remote transfer failed"

Diagnosis:

# Test storage box connectivity
ssh -p 23 [storage-box-user]@[storage-box-host]

# Check WHM backup logs
tail -100 /usr/local/cpanel/logs/cpbackup/[date].log

# Test SFTP transfer manually
sftp -P 23 [storage-box-user]@[storage-box-host]
put test.txt

Resolution: - Verify SSH keys are configured correctly - Check storage box authentication - Verify storage box has capacity - Check firewall rules (port 23/SSH) - Reconfigure backup destination in WHM

3. Database Backup Fails

Symptoms: - Account backup succeeds but database missing - Error: "MySQL dump failed"

Diagnosis:

# Check MySQL service
systemctl status mysql

# Test database access
mysql -u root -p -e "SHOW DATABASES;"

# Check database size
mysql -e "SELECT table_schema AS 'Database',
           ROUND(SUM(data_length + index_length) / 1024 / 1024, 2) AS 'Size (MB)'
           FROM information_schema.tables
           GROUP BY table_schema;"

Resolution: - Verify MySQL is running - Check database permissions - Repair corrupted databases: mysqlcheck --repair --all-databases - Check disk space for temp files during dump - Manually dump problem database to test

4. Backup Runs Too Long

Symptoms: - Backup still running after 6+ hours - Server performance degraded during backup

Diagnosis:

# Check backup process
ps aux | grep backup

# Check I/O wait
top
# Look for high %wa (I/O wait)

# Check backup logs for slow accounts
tail -f /usr/local/cpanel/logs/cpbackup/[date].log

Resolution: - Identify large accounts causing delays - Consider excluding large log directories - Adjust backup compression level - Split backups across different times - Upgrade storage/network bandwidth

5. Backup Restoration Fails

Symptoms: - Restore process hangs or fails - Error: "Invalid archive" or "Corrupted backup"

Diagnosis:

# Test backup file integrity
tar -tzf /path/to/backup.tar.gz | head -20

# Check file size (compare to expected)
ls -lh /path/to/backup.tar.gz

# Try extracting to temp location
mkdir /tmp/test_restore
cd /tmp/test_restore
tar -xzf /path/to/backup.tar.gz

Resolution: - Download backup again (may be transfer corruption) - Try older backup if recent backup corrupted - Extract backup manually and restore components individually - Check storage box for backup file integrity - Report to Hetzner if storage box issue

Emergency Contact

Hetzner Support: - Web: https://www.hetzner.com/support - Emergency: Available 24/7 via support portal - Storage Box Issues: Create support ticket

cPanel Support: - Technical Support: Via WHM support portal - Documentation: https://docs.cpanel.net

Internal Escalation: - Primary: System Administrator - Backup: See Contacts

Security Considerations

Backup Data Protection

Encryption Status: - ✅ Backups are encrypted in transit (SSH/SFTP transport encryption) - 🔜 Backups to be encrypted at rest using GPG (AES-256) - see implementation plan below - 🔐 Storage box access protected by SSH keys + password - 🔐 Storage box access restricted to MDHosting infrastructure

Access Control: - ✅ Storage box accessible only via SSH key authentication - ✅ SSH keys stored securely in password manager - ✅ Storage box credentials not shared - ✅ Regular review of storage box access logs

GDPR Compliance: - Backups contain client personal data - Retention periods align with data retention policies - Right to erasure: Client account deletion includes backup removal - Encryption at rest (planned) will provide additional technical safeguard per Article 32 - See GDPR Compliance for data subject rights

Backup Encryption Implementation (Planned)

Implementation Status

This section documents the planned backup encryption implementation using GPG and YubiKey hardware tokens. Implementation is scheduled for Q1 2026 to address security audit finding C-2.

MDHosting will use GPG encryption with hardware-backed key storage (YubiKey) to protect backup data at rest. This ensures that even if the storage box is compromised, backup data remains protected.

Architecture Overview

graph LR
    subgraph "EU1 Production Server"
        CPANEL[cPanel Backup Engine]
        HOOK[Post-Backup Hook]
        PUBKEY[GPG Public Key]
    end

    subgraph "Hetzner Storage Box"
        ENCRYPTED[Encrypted Backups<br/>.tar.gz.gpg]
    end

    subgraph "Offline Key Storage"
        YUBIKEY1[YubiKey 5 NFC<br/>Primary]
        YUBIKEY2[YubiKey 5 NFC<br/>Backup]
    end

    CPANEL -->|Creates| HOOK
    HOOK -->|Encrypts with| PUBKEY
    HOOK -->|Transfers| ENCRYPTED
    YUBIKEY1 -.->|Decrypts for Recovery| ENCRYPTED
    YUBIKEY2 -.->|Backup Decryption| ENCRYPTED

    classDef server fill:#3498db,stroke:#2c3e50,stroke-width:2px,color:#fff
    classDef storage fill:#27ae60,stroke:#2c3e50,stroke-width:2px,color:#fff
    classDef key fill:#e74c3c,stroke:#2c3e50,stroke-width:2px,color:#fff

    class CPANEL,HOOK,PUBKEY server
    class ENCRYPTED storage
    class YUBIKEY1,YUBIKEY2 key

Encryption Specifications

Component	Specification
Algorithm	GPG (GnuPG) with AES-256
Key Type	RSA 4096-bit
Key Storage	YubiKey 5 NFC (hardware token)
Public Key Location	EU1 server (`/root/.gnupg/`)
Private Key Location	YubiKey only (never on server)
Backup Key	Second YubiKey stored in secure location

YubiKey GPG Key Management

MDHosting uses two YubiKey 5 NFC devices for backup encryption key management:

Primary YubiKey: Day-to-day recovery operations
Backup YubiKey: Stored securely off-site for disaster recovery

Critical Security Control

The GPG private key exists only on the YubiKey hardware tokens. It is never stored on any server, computer, or cloud service. This ensures that backup decryption requires physical possession of a YubiKey.

Initial YubiKey Setup

Prerequisites: - 2x YubiKey 5 NFC devices - Workstation with GnuPG 2.2+ installed - USB connection (not NFC for initial setup)

Step 1: Configure YubiKey PIN and Admin PIN

# Insert YubiKey and configure
gpg --card-edit

# Inside gpg card interface:
gpg/card> admin
gpg/card> passwd

# Set the following (store in password manager):
# - PIN (6-8 digits): Used for daily operations
# - Admin PIN (8 digits): Used for key management
# - Reset Code (8 digits): Emergency PIN reset

gpg/card> quit

Step 2: Generate GPG Key Directly on YubiKey

# Generate key on the YubiKey (key never leaves hardware)
gpg --card-edit

gpg/card> admin
gpg/card> generate

# When prompted:
# - Make off-card backup: NO (key stays on YubiKey only)
# - Key expires: 0 (does not expire)
# - Key size: 4096 bits (RSA)
# - Real name: MDHosting Backup Encryption
# - Email: admin@mdhosting.co.uk
# - Comment: Primary Key

gpg/card> quit

Step 3: Export Public Key

# Export public key for use on servers
gpg --export --armor admin@mdhosting.co.uk > mdhosting-backup-pubkey.asc

# Verify the key
gpg --list-keys admin@mdhosting.co.uk

Step 4: Configure Second YubiKey (Backup)

Repeat Steps 1-3 with the second YubiKey, using the same identity but generating a new key pair. Both YubiKeys will be able to decrypt backups encrypted to either key.

# For the backup YubiKey, add to the same keyring
# Generate on second YubiKey with same identity
gpg --card-edit
# ... same process as above ...

# Export both public keys
gpg --export --armor admin@mdhosting.co.uk > mdhosting-backup-pubkeys.asc

YubiKey Storage Locations

YubiKey	Location	Access
Primary	Secure location at administrator's premises	Day-to-day recovery
Backup	Off-site secure storage (bank safe deposit or trusted location)	Disaster recovery only

Key Separation

Never store both YubiKeys in the same location. The backup YubiKey should be geographically separated to protect against theft, fire, or other localised disasters.

Server-Side Encryption Configuration

Installing GPG Public Key on EU1

# SSH to EU1
ssh eu1.cp

# Create GPG directory if not exists
mkdir -p /root/.gnupg
chmod 700 /root/.gnupg

# Import the public key (copy from workstation)
cat > /root/.gnupg/mdhosting-backup-pubkey.asc << 'EOF'
-----BEGIN PGP PUBLIC KEY BLOCK-----
[PUBLIC KEY CONTENT - Import from YubiKey export]
-----END PGP PUBLIC KEY BLOCK-----
EOF

# Import into GPG keyring
gpg --import /root/.gnupg/mdhosting-backup-pubkey.asc

# Trust the key for encryption
gpg --edit-key admin@mdhosting.co.uk trust
# Select: 5 (ultimate trust)
# Confirm: y

# Verify key is available
gpg --list-keys admin@mdhosting.co.uk

Post-Backup Encryption Script

Create the encryption script at /usr/local/bin/encrypt-backups.sh:

#!/bin/bash
#
# MDHosting Backup Encryption Script
# Encrypts cPanel backups using GPG before transfer to storage box
#
# Location: /usr/local/bin/encrypt-backups.sh
# Permissions: chmod 700 /usr/local/bin/encrypt-backups.sh
# Owner: root:root
#

set -euo pipefail

# Configuration
BACKUP_DIR="/backup"
GPG_RECIPIENT="admin@mdhosting.co.uk"
LOG_FILE="/var/log/backup-encryption.log"
LOCK_FILE="/var/run/backup-encryption.lock"

# Logging function
log() {
    echo "$(date '+%Y-%m-%d %H:%M:%S') - $1" >> "$LOG_FILE"
}

# Check for lock file (prevent concurrent runs)
if [[ -f "$LOCK_FILE" ]]; then
    log "ERROR: Lock file exists. Another encryption process may be running."
    exit 1
fi

# Create lock file
trap "rm -f $LOCK_FILE" EXIT
touch "$LOCK_FILE"

log "Starting backup encryption process"

# Count files to process
TOTAL_FILES=$(find "$BACKUP_DIR" -name "*.tar.gz" -type f ! -name "*.tar.gz.gpg" 2>/dev/null | wc -l)
PROCESSED=0
ERRORS=0

log "Found $TOTAL_FILES unencrypted backup files"

# Process each unencrypted backup
find "$BACKUP_DIR" -name "*.tar.gz" -type f ! -name "*.tar.gz.gpg" | while read -r backup_file; do
    # Skip if already has encrypted version
    if [[ -f "${backup_file}.gpg" ]]; then
        log "SKIP: Encrypted version exists for $(basename "$backup_file")"
        continue
    fi

    log "ENCRYPTING: $(basename "$backup_file")"

    # Encrypt the backup
    if gpg --encrypt \
           --recipient "$GPG_RECIPIENT" \
           --trust-model always \
           --compress-algo none \
           --output "${backup_file}.gpg" \
           "$backup_file" 2>> "$LOG_FILE"; then

        # Verify encrypted file exists and has content
        if [[ -f "${backup_file}.gpg" ]] && [[ -s "${backup_file}.gpg" ]]; then
            # Securely remove unencrypted version
            shred -u "$backup_file" 2>/dev/null || rm -f "$backup_file"
            log "SUCCESS: Encrypted and removed $(basename "$backup_file")"
            ((PROCESSED++)) || true
        else
            log "ERROR: Encryption produced empty file for $(basename "$backup_file")"
            rm -f "${backup_file}.gpg"
            ((ERRORS++)) || true
        fi
    else
        log "ERROR: GPG encryption failed for $(basename "$backup_file")"
        ((ERRORS++)) || true
    fi
done

log "Backup encryption complete. Processed: $PROCESSED, Errors: $ERRORS"

# Send alert if errors occurred
if [[ $ERRORS -gt 0 ]]; then
    echo "Backup encryption completed with $ERRORS errors. Check $LOG_FILE for details." | \
        mail -s "[MDHosting] Backup Encryption Errors" admin@mdhosting.co.uk
fi

exit 0

Set permissions:

chmod 700 /usr/local/bin/encrypt-backups.sh
chown root:root /usr/local/bin/encrypt-backups.sh

Configuring cPanel Backup Hook

Register the encryption script to run after cPanel backups complete:

# Create hooks directory if not exists
mkdir -p /var/cpanel/hooks/backup

# Create hook configuration
cat > /var/cpanel/hooks/backup/post_backup.yaml << 'EOF'
---
post_backup:
  - executable: /usr/local/bin/encrypt-backups.sh
    stages:
      - post
    escalate_privileges: 1
EOF

# Register the hook
/usr/local/cpanel/bin/manage_hooks add script /usr/local/bin/encrypt-backups.sh \
    --category Backup \
    --event post_backup \
    --stage post

# Verify hook is registered
/usr/local/cpanel/bin/manage_hooks list

Cron Job (Alternative Method)

If cPanel hooks are problematic, use a cron job instead:

# Add to root crontab
crontab -e

# Run encryption 2 hours after backup window (adjust time as needed)
0 5 * * * /usr/local/bin/encrypt-backups.sh >> /var/log/backup-encryption.log 2>&1

Backup Decryption Procedures

Standard Recovery (With YubiKey)

Prerequisites: - YubiKey with GPG private key - Workstation with GnuPG installed - Network access to storage box

Step 1: Connect YubiKey

# Insert YubiKey into workstation USB port

# Verify YubiKey is detected
gpg --card-status

# Should show:
# Application ID: [YubiKey ID]
# Key attributes: rsa4096

Step 2: Download Encrypted Backup

# SSH to storage box or use SFTP
sftp [storage-box-credentials]@[storage-box-host]

# Download required backup
get /backups/daily/cpmove-accountname-20260114.tar.gz.gpg

# Exit SFTP
exit

Step 3: Decrypt Backup

# Decrypt the backup (YubiKey PIN will be requested)
gpg --decrypt cpmove-accountname-20260114.tar.gz.gpg > cpmove-accountname-20260114.tar.gz

# Enter YubiKey PIN when prompted

# Verify decrypted file
tar -tzf cpmove-accountname-20260114.tar.gz | head

Step 4: Transfer and Restore

# Transfer to target server
scp cpmove-accountname-20260114.tar.gz eu1.cp:/home/

# SSH to server and restore
ssh eu1.cp
/scripts/restorepkg /home/cpmove-accountname-20260114.tar.gz

# Clean up
rm /home/cpmove-accountname-20260114.tar.gz

Emergency Recovery (Remote Decryption)

If the administrator is remote but has physical access to a YubiKey:

Option A: Decrypt Locally, Upload

# On local workstation with YubiKey
# 1. Download encrypted backup via SFTP
# 2. Decrypt locally
# 3. Upload decrypted backup to server
# 4. Restore via cPanel
# 5. Delete local copies securely

Option B: Temporary Key Export (Emergency Only)

Use Only in Genuine Emergency

This procedure temporarily exports a decryption capability to a server. Only use when physical YubiKey access is impossible and recovery is critical.

# This is NOT recommended for normal operations
# Document any use of this procedure in incident log

Key Recovery Procedures

Lost Primary YubiKey

If the primary YubiKey is lost or damaged:

Immediately retrieve backup YubiKey from secure storage
Use backup YubiKey for any required recovery operations
Order replacement YubiKey
Generate new key pair on replacement YubiKey
Add new public key to server encryption configuration
Re-encrypt recent backups with both keys
Return backup YubiKey to secure storage
Document incident

Lost Both YubiKeys

Critical Incident

Loss of both YubiKeys means permanent loss of access to encrypted backups. This is by design - it ensures that compromise of infrastructure alone cannot expose backup data.

Recovery Options: 1. Recent data: Restore from production server (if still operational) 2. Historical data: Unrecoverable - encrypted backups cannot be decrypted

Prevention: - Store YubiKeys in geographically separate locations - Consider a third YubiKey in a bank safe deposit box - Regularly verify both YubiKeys are functional

YubiKey PIN Lockout

After 3 incorrect PIN attempts, the YubiKey locks. After 3 incorrect Admin PIN attempts, the YubiKey requires a factory reset (destroying the key).

Recovery: 1. Use Reset Code (if configured) to reset PIN 2. If Reset Code fails, use backup YubiKey 3. If backup YubiKey also locked, follow "Lost Both YubiKeys" procedure

Encryption Verification and Testing

Monthly Verification Checklist

Check	Command/Action	Expected Result
Encrypted files exist	`ls /backups/daily/*.gpg`	List of .gpg files
No unencrypted files	`ls /backups/daily/*.tar.gz`	No files (or very recent only)
GPG public key valid	`gpg --list-keys admin@mdhosting.co.uk`	Key listed, not expired
YubiKey functional	`gpg --card-status`	Card status displayed
Test decryption	Decrypt a test backup	Successful decryption

Quarterly Decryption Test

# Select a random backup for test decryption
TEST_BACKUP=$(ls /backups/daily/*.gpg | shuf -n 1)

# Download to workstation
sftp> get "$TEST_BACKUP"

# Decrypt with YubiKey
gpg --decrypt "$(basename $TEST_BACKUP)" > test-restore.tar.gz

# Verify archive integrity
tar -tzf test-restore.tar.gz > /dev/null && echo "Archive valid"

# Clean up
rm test-restore.tar.gz "$(basename $TEST_BACKUP)"

# Document test in maintenance log

Article 32 Compliance

Backup encryption satisfies GDPR Article 32 requirements for "appropriate technical measures":

Requirement	Implementation
Encryption of personal data	AES-256 via GPG
Pseudonymisation	Backup filenames contain account names only
Confidentiality	Hardware-backed key storage (YubiKey)
Integrity	GPG signatures verify data integrity
Availability	Dual YubiKey redundancy
Resilience	Encrypted backups survive infrastructure compromise

Breach Notification Impact

With encrypted backups: - Encrypted backup breach: May not require notification if encryption is "state of the art" (GDPR Recital 83) - Key compromise + backup breach: Full notification required - Documentation: Maintain evidence of encryption implementation

Data Subject Rights

Encrypted backups do not change data subject rights:

Right	Process
Access	Decrypt relevant backup, extract data
Erasure	Delete encrypted backup files (same as before)
Portability	Decrypt and export account data

Audit Trail

Maintain logs of: - Encryption script execution (/var/log/backup-encryption.log) - Decryption operations (GPG logs) - YubiKey access (physical access log) - Key management changes

Backup Retention and Data Deletion

Account Deletion Process: 1. Remove account from production server 2. Remove account backups from storage box (all retention periods) 3. Document deletion in client records 4. Confirm deletion to client if requested (GDPR right to erasure)

Command to Delete Account Backups:

# SSH to storage box
ssh [storage-box]

# Remove all backups for deleted account (encrypted format)
rm /backups/daily/cpmove-[account]-*.tar.gz.gpg
rm /backups/weekly/cpmove-[account]-*.tar.gz.gpg
rm /backups/monthly/cpmove-[account]-*.tar.gz.gpg

# Also remove any unencrypted backups (legacy/transition period)
rm /backups/daily/cpmove-[account]-*.tar.gz 2>/dev/null || true
rm /backups/weekly/cpmove-[account]-*.tar.gz 2>/dev/null || true
rm /backups/monthly/cpmove-[account]-*.tar.gz 2>/dev/null || true

Disaster Recovery Planning

Business Continuity Scenarios

Scenario 1: Single Server Failure

Event: EU1 hosting server hardware failure

Impact: - All client websites offline - Email services disrupted - Client data at risk

Recovery Plan: - Provision new Hetzner server (1-2 hours) - Install cPanel (1 hour) - Restore all accounts from storage box (4-8 hours) - Update DNS if IP changed (24-48 hours for propagation) - Total RTO: 8-12 hours (24-48 hours for full DNS propagation)

Mitigation: - Regular backup testing - Documented recovery procedures - Pre-tested server provisioning process

Scenario 2: Storage Box Failure

Event: Hetzner Storage Box failure or data loss

Impact: - Loss of backup history - Cannot restore from backups - Production data still intact

Recovery Plan: - Contact Hetzner support immediately - Trigger immediate full backup to alternative location - Provision alternative backup storage (AWS S3, Backblaze B2) - Configure dual backup destinations - Total RTO: N/A (production not affected)

Mitigation: - Hetzner Storage Box has built-in redundancy - Consider secondary backup location for critical data - Regular backup testing ensures early detection

Scenario 3: Complete Datacentre Failure

Event: Hetzner Germany datacentre unavailable

Impact: - All servers offline (hosting + DNS + backups) - Complete service outage - Maximum business impact

Recovery Plan: - Provision servers at alternative provider (2-4 hours) - Access storage box backups (if accessible) - If storage box unavailable, download from recent local backups - Restore all accounts (8-12 hours) - Update nameserver records at internet.bs (1-24 hours) - Total RTO: 16-24 hours

Mitigation: - Future: Consider geographic redundancy (UK backup location) - Document alternative provider options (AWS, DigitalOcean) - Maintain offline copy of critical documentation - Regular RTO/RPO testing

Scenario 4: Ransomware Attack

Event: Ransomware encrypts production server data

Impact: - Client data encrypted and inaccessible - Potential backup infection if ransomware spreads - Ransom demand

Recovery Plan: - DO NOT pay ransom - Immediately isolate affected servers (network disconnect) - Verify backups not encrypted (storage box immutable) - Provision clean server - Restore from most recent clean backup - Conduct malware analysis to identify entry point - Total RTO: 8-12 hours

Mitigation: - Regular security updates - Fail2Ban and CSF firewall protection - Wazuh SIEM monitoring (planned) - ClamAV malware scanning - Regular backup testing ensures clean restore capability

RTO/RPO Summary

Recovery Scenario	RPO (Data Loss)	RTO (Recovery Time)	Priority
Single file restore	24 hours	15-30 minutes	High
Database restore	24 hours	15-45 minutes	High
Full account restore	24 hours	1-4 hours	High
Single server failure	24 hours	8-12 hours	Critical
Complete disaster recovery	24 hours	16-24 hours	Critical

Recovery Priority Order: 1. eu1.cp (client hosting) - Critical 2. ns1/ns2 (DNS) - High 3. Non-critical services - Medium

Future Improvements

Short-Term (Next 6 Months)

Implement quarterly backup restore testing
Document exact backup timing (identify cron schedule)
Create backup monitoring dashboard
Test backup encryption (GPG)
Verify storage box capacity planning for 1-2 year growth

Medium-Term (6-12 Months)

Implement secondary backup location (AWS S3 or Backblaze B2)
Automate backup integrity testing
Create client-facing backup status portal
Document ApisCP backup migration plan
Implement backup reporting metrics

Long-Term (12+ Months)

Evaluate continuous replication solutions
Consider geographic redundancy (UK backup storage)
Implement immutable backups (ransomware protection)
Automate disaster recovery testing
Evaluate backup solutions for post-ApisCP migration

Appendices

A. Quick Reference Commands

Check Backup Status:

# View recent backup logs
tail -100 /usr/local/cpanel/logs/cpbackup/$(date +%Y-%m-%d).log

# Check running backup processes
ps aux | grep backup

# View backup configuration
/usr/local/cpanel/bin/backup_config_get

Manual Backup Trigger:

# Trigger full backup manually (use with caution)
/scripts/cpbackup --force

# Backup single account
/scripts/pkgacct [username]

Storage Box Access:

# SSH to storage box
ssh -p 23 [storage-box-user]@[storage-box-host]

# SFTP to storage box
sftp -P 23 [storage-box-user]@[storage-box-host]

# List backups
ssh -p 23 [storage-box-user]@[storage-box-host] ls -lh /backups/daily/

Restore Commands:

# Restore account from backup
/scripts/restorepkg /path/to/cpmove-[account].tar.gz

# Restore account with new username
/scripts/restorepkg --newuser=[newname] /path/to/backup.tar.gz

B. Backup Testing Checklist

Monthly Backup Test: - [ ] Review backup success emails (last 7 days) - [ ] Check WHM backup logs for errors - [ ] SSH to storage box and verify backups exist - [ ] Check storage box capacity usage - [ ] Download and test one random account backup - [ ] Verify 5-10 random files from backup - [ ] Document test results

Quarterly Backup Test: - [ ] Complete all monthly tests - [ ] Restore one database to test database - [ ] Verify database integrity - [ ] Perform full test account restoration - [ ] Verify all account components - [ ] Time the restoration process - [ ] Document RTO achieved

Annual Disaster Recovery Test: - [ ] Complete all monthly and quarterly tests - [ ] Provision test server - [ ] Perform simulated full server recovery - [ ] Time entire recovery process - [ ] Test all services on recovered server - [ ] Document lessons learned - [ ] Update disaster recovery procedures - [ ] Generate annual backup report

C. Backup Retention Examples

Example Account Timeline:

Account: exampleclient.com
Created: 2024-01-15
Current Date: 2025-01-07

Available Backups:
Daily (Last 7 days):
  - 2025-01-07 (today)
  - 2025-01-06
  - 2025-01-05
  - 2025-01-04
  - 2025-01-03
  - 2025-01-02
  - 2025-01-01

Weekly (Last 4 weeks):
  - 2024-12-31 (Week 1)
  - 2024-12-24 (Week 2)
  - 2024-12-17 (Week 3)
  - 2024-12-10 (Week 4)

Monthly (Last 3 months):
  - 2024-12-01 (Month 1)
  - 2024-11-01 (Month 2)
  - 2024-10-01 (Month 3)

Total Available Restore Points: 14 backups
Earliest Restore Point: 2024-10-01 (3 months ago)

D. Contact Information for Backup Issues

Internal: - System Administrator: (See Contacts) - Email Alerts: admin@mdhosting.co.uk

External Vendors: - Hetzner Support: https://www.hetzner.com/support (Storage Box issues) - cPanel Support: https://docs.cpanel.net (Backup system issues)

Emergency Procedures: - See Incident Response for escalation

Last updated: January 2026 Next review: Before ApisCP migration commencement

Backup & Recovery Procedures

Overview

Current Backup Infrastructure

Backup Architecture

Backup Solution

What Gets Backed Up

eu1.cp (Primary Hosting Server)

DNS Servers (ns1, ns2)

Backup Schedule & Retention

Backup Timing

Retention Policy

Storage Management

Backup Verification

Automated Monitoring

Manual Verification

Recovery Procedures

Recovery Scenarios

1. File Recovery (Individual Files/Folders)

2. Database Recovery

3. Full Account Recovery

4. Full Server Recovery (Disaster Recovery)

Phase 1: Server Preparation (1-2 hours)

Phase 2: Backup Restoration (2-8 hours)

Phase 3: Service Verification (1-2 hours)

Phase 4: Post-Recovery (1-2 hours)

Disaster Recovery Checklist

Client Self-Service Restore

cPanel Backup Interface

Backup Configuration

cPanel/WHM Backup Settings

Storage Box Configuration

Backup Testing Schedule

Monthly Testing

Quarterly Testing

Annual Testing

Troubleshooting

Common Backup Issues

1. Backup Fails with "Disk Space Full"

2. Backup Transfer Fails to Storage Box

3. Database Backup Fails

4. Backup Runs Too Long

5. Backup Restoration Fails

Emergency Contact

Security Considerations

Backup Data Protection

Backup Encryption Implementation (Planned)

Architecture Overview

Encryption Specifications

YubiKey GPG Key Management

Initial YubiKey Setup

YubiKey Storage Locations

Server-Side Encryption Configuration

Installing GPG Public Key on EU1

Post-Backup Encryption Script

Configuring cPanel Backup Hook

Cron Job (Alternative Method)

Backup Decryption Procedures

Standard Recovery (With YubiKey)

Emergency Recovery (Remote Decryption)

Key Recovery Procedures

Lost Primary YubiKey

Lost Both YubiKeys

YubiKey PIN Lockout

Encryption Verification and Testing

Monthly Verification Checklist

Quarterly Decryption Test

GDPR Compliance for Encrypted Backups

Article 32 Compliance

Breach Notification Impact

Data Subject Rights

Audit Trail

Backup Retention and Data Deletion

Disaster Recovery Planning

Business Continuity Scenarios

Scenario 1: Single Server Failure

Scenario 2: Storage Box Failure

Scenario 3: Complete Datacentre Failure

Scenario 4: Ransomware Attack

RTO/RPO Summary

Future Improvements