ddev providers

# ddev providers

---

## Topics

- What are "ddev providers"? 
    - Stanzas 
    - Command 
- Provider Examples 
    - Local 
    - Webserver/SSH 
    - GitLab 
- Additional 
    - GitLab Provider Example 
    - Filefill Config 
    - Hooks for comfort 
---

## What are "ddev providers"?

- Webserver <--> ddev 
- Data & Files 
- <span>Recipies (yaml) in `.ddev/providers`</span> 
    - <span>`ddev pull <filename>`</span> 
- [Documentation](https://ddev.readthedocs.io/en/stable/users/providers/) 
- <span>`host` vs `web`</span> 
- <span>Use [ext:filefill](https://extensions.typo3.org/extension/filefill) for Files</span>

----

### ddev provider stanzas

- environment_variables
    - Provides environment variables to the web container. 
- auth_command
    - Validate SSH key or env variables etc. 
- db_pull_command (optional)
    - Its job is to create a gzipped database dump in /var/www/html/.ddev/.downloads/db.sql.gz. 
- db_import_command (optional)
    - This is for advanced usages like multiple databases. The default behavior only imports a single database into the db database. 
- files_pull_command
    - Its job is to copy the files from upstream to /var/www/html/.ddev/.downloads/files. 
- files_import_command (optional)
    - A script that imports the downloaded files. 
- db_push_command
    - Its job is to take a gzipped database dump from /var/www/html/.ddev/.downloads/db.sql.gz and load it on the hosting provider. 
- files_push_command
    - Its job is to copy the files from the project’s user-files directories ($DDEV_FILES_DIRS) to the correct places on the upstream provider.

----

### Command structure

Pulling `ddev pull <filename>`

Options:
- --skip-db
- --skip-files
- --skip-import

---

## Local

Populate database and parts of fileadmin from local file committed to the project

### Pros

+ Fast
+ Offline available

</div>

### Cons

- Manual update of dump
- Dump mostly very old
- Enlarges git project

</div>
</div>

----

#### `.ddev/providers/local.yaml`

```yaml[3-4|6-10|12-18]
# TYPO3 local data provider

db_import_command:
  command: |
    set -eu -o pipefail
    gzip .data/db.sql.gz | ddev mysql db
  service: host

files_import_command:
  command: |
    # set -x   # You can enable bash debugging output by uncommenting
    set -eu -o pipefail
    mkdir -p ${DDEV_DOCROOT}/fileadmin
    unzip /var/www/html/.data/files.zip -d ${DDEV_DOCROOT}/fileadmin
  service: web
```

```bash
ddev pull local
```

---

## Webserver/SSH

Populate database and parts of fileadmin from the webserver directly via ssh & rsync

### Pros

+ Latest data & files
+ No dumps in git

</div>

### Cons

- SSH Access to webserver

</div>
</div>

----

### `.ddev/providers/production.yaml`

```yaml[16-23|25-28|30-38|40-48|50-55]
# TYPO3 webserver data provider

# This will pull a database and files from a TYPO3 instance using the helhum/typo3-console. It operates inside the web
# container and uses ssh, so you need to run `ddev auth ssh` first to copy your ssh key into the ddev container.

# Note that only the "fileadmin/form_definitions" folder is fetched via files pull command.
# The majority of the files should be pulled on use with ext:filefill. This way you save a lot of diskspace.
# If you need to fetch more files, have a look at this comment on how to rsync easy with
# --exclude="*" multiple directories: https://stackoverflow.com/a/11111793

#####################
### CONFIGURATION ###
#####################
# In the environment_variables section you need to define the server as user@host and the path to your TYPO3 instance.
# The rysnc includes/excludes and the database excluded tables need to be configured in the commands directly.

environment_variables:
  server: user@example.com
  path: /var/www/vhosts/example.com
  php: php
  typo3_console: vendor/bin/typo3cms
  port: 22
  webroot: public

auth_command:
  command: |
    set -eu -o pipefail
    ssh-add -l >/dev/null || ( echo "Please 'ddev auth ssh' before running this command." && exit 1 )

db_pull_command:
  command: |
    # set -x # You can enable bash debugging output by uncommenting
    set -eu -o pipefail
    # Temporarely define excludeTables inside command. Should be moved to yaml config, but this throws errors.
    excludeTables=(cf_* cache_* sfc_* [bf]e_sessions [bf]e_users index_* sys_file_processedfile sys_lockedrecords sys_log tx_extensionmanager_* tx_staticfilecache_queue tx_solr_indexqueue_item tx_webp_failed zzz_*)
    for i in "${excludeTables[@]}"; do exlcudes=${exlcudes:+$exlcudes }"-e '$i'"; done
    ssh ${server} -A -p ${port} -o LogLevel=error "cd ${path}; ${php} ${typo3_console} database:export -c Default ${exlcudes} | gzip" > /var/www/html/.ddev/.downloads/db.sql.gz
  service: web

files_pull_command:
  command: |
    # set -x   # You can enable bash debugging output by uncommenting
    set -eu -o pipefail
    ls /var/www/html/.ddev >/dev/null # This just refreshes stale NFS if possible
    mkdir -p /var/www/html/.ddev/.downloads
    pushd /var/www/html/.ddev/.downloads >/dev/null
    rsync -azrL --include="form_definitions" --include="form_definitions/***" --exclude="*" -e "ssh -p ${port} -o LogLevel=error" "${server}:${path}/${webroot}/fileadmin/" fileadmin/
  service: web

files_import_command:
  command: |
    # set -x   # You can enable bash debugging output by uncommenting
    set -eu -o pipefail
    cp -r /var/www/html/.ddev/.downloads/fileadmin ${webroot}/ && rm -r /var/www/html/.ddev/.downloads/fileadmin
  service: web
```

```bash
ddev auth ssh && ddev pull production
```

---

## GitLab

Populate database and parts of fileadmin from GitLab Package Registry

### Pros

+ One-Click update data & files
+ No dumps in git
+ No SSH Access
+ Access through GitLab

</div>

### Cons

- Initial CI Setup
- Setup Personal Access Token

</div>
</div>

----

### GitLab configuration

1. Enable "Package registry" for your project 
    - Settings > General > Visibility, project features, permissions 
2. Set "Number of duplicate assets to keep" to 1 
    - Settings > Packages and registries 
3. Define GitLab CI Task 
    1. Create dumps 
    2. Push to Package registry

----

#### `.gitlab-ci.yml`

```yaml[5-8|12-17|18-21]
dump-production-for-ddev:
  image: $IMAGE_PHP
  stage: .pre
  when: manual
  variables:
    GIT_STRATEGY: none
    GITLAB_RELEASE_VERSION: "1.0.0"
    PACKAGE_REGISTRY_URL: "${CI_API_V4_URL}/projects/${CI_PROJECT_ID}/packages/generic/site"
  before_script:
    - *setup_ssh
    - apk add rsync --update
  script:
    - ssh -4 $SSH_USER@$SSH_SERVER "$PHP $SSH_REMOTE_PATH/vendor/bin/typo3cms database:export -c Default -e 'cache_*' -e '[bf]e_sessions' -e '[bf]e_users' -e 'index_*' -e 'sys_log' -e 'sys_lockedrecords' -e 'sys_file_processedfile' -e 'tx_webp_failed' -e 'tx_extensionmanager_*' -e 'tx_staticfilecache_queue' -e 'tx_solr_indexqueue_item' -e 'zzz_*' | gzip -9 -c" > ./db.sql.gz
    - mkdir files
    - rsync -avz --include="user" --include="user/***" --exclude="*" $SSH_USER@$SSH_SERVER:$SSH_REMOTE_PATH/public/fileadmin/ ./files/fileadmin
    - rsync -avz --include="tx_junithcore" --include="tx_junithcore/***" --exclude="*" $SSH_USER@$SSH_SERVER:$SSH_REMOTE_PATH/public/uploads/ ./files/uploads
    - zip -r files.zip files/*
    # Make sure to set "Number of duplicate assets to keep" to 1 in your GitLab project settings -> Packages and registries
    - |
      curl --fail-with-body --header "JOB-TOKEN: ${CI_JOB_TOKEN}" --upload-file ./files.zip ${PACKAGE_REGISTRY_URL}/${GITLAB_RELEASE_VERSION}/files.zip
      curl --fail-with-body --header "JOB-TOKEN: ${CI_JOB_TOKEN}" --upload-file ./db.sql.gz ${PACKAGE_REGISTRY_URL}/${GITLAB_RELEASE_VERSION}/db.sql.gz

.setup_ssh: &setup_ssh
  # Run ssh-agent (inside the build environment)
  - eval $(ssh-agent -s)

# Add the SSH key stored in SSH_PRIVATE_KEY variable to the agent store
  - mkdir -p ~/.ssh
  - chmod 700 ~/.ssh
  - echo "$SSH_PRIVATE_KEY" | ssh-add -
  - '[[ -f /.dockerenv ]] && echo -e "Host *\n\tStrictHostKeyChecking no\n\tControlMaster auto\n\tControlPath /tmp/ssh_mux_%h_%p_%r\n\tControlPersist 600\n\n" > ~/.ssh/config'
```

----

### Pesonal Access Token

1. Create a personal access token 
    - User > Preferences > Access tokens 
    - Expiration date: none 
    - Select scopes: read_api 
2. Add PAT as environment variable to ddev globally 
    - GITLAB_COM_PAT 
    - GITLAB_MYCOMPANY_PAT 
    - ...

```bash
ddev config global --web-environment-add="GITLAB_COM_PAT=<TOKEN>"
```

----

#### `.ddev/providers/gitlab.yaml`

```yaml[14-18|20-29|31-38|40-48|50-56]
# TYPO3 gitlab data provider

# This will pull the database and files from GitLab packages. You may need to execute the dump-job first in GitLab.

#####################
### CONFIGURATION ###
#####################
# In the environment_variables section you need to define the GitLab project id and the package tag version.
# The webroot is relative to your ddev entry point (see working_dir in .ddev/config.yaml).

environment_variables:
  GITLAB_DOMAIN: gitlab.com
  PACKAGE_NAME: site
  PACKAGE_VERSION: 1.0.0
  PROJECT_ID: 12345

auth_command:
  command: |
    set -eu -o pipefail
    echo "Pulling dump from GitLab..."
    if [ -z "${GITLAB_COM_PAT:-}" ]; then echo "Please make sure you have set the GITLAB_COM_PAT environment variable. You need to create a personal access token with scope 'read_api' here: https://${GITLAB_DOMAIN}/-/user_settings/personal_access_tokens . Add it globally to ddev: 'ddev config global --web-environment=\"GITLAB_COM_PAT=<TOKEN>\"' and restart the project." && exit 1; fi
    VALIDATION_RESPONSE=$(curl --silent --write-out "%{http_code}" --output /dev/null --request GET --header "PRIVATE-TOKEN: ${GITLAB_COM_PAT}" --url "https://${GITLAB_DOMAIN}/api/v4/personal_access_tokens/self")
    [ "$VALIDATION_RESPONSE" -eq 401 ] && echo "Error: Unauthorized. The provided GitLab Personal Access Token is invalid, does not have the required permissions or is expired. Please check here: https://${GITLAB_DOMAIN}/-/user_settings/personal_access_tokens" && exit 1
    [ "$VALIDATION_RESPONSE" -eq 404 ] && echo "Error: Not Found. The token does not exist." && exit 1
    [ "$VALIDATION_RESPONSE" -ne 200 ] && echo "Error: Unexpected response from GitLab API [HTTP $VALIDATION_RESPONSE]." && exit 1
    echo "GitLab authentication: OK"

db_pull_command:
  command: |
    # set -x # You can enable bash debugging output by uncommenting
    set -eu -o pipefail
    mkdir -p /var/www/html/.ddev/.downloads
    pushd /var/www/html/.ddev/.downloads >/dev/null
    curl --header "PRIVATE-TOKEN: ${GITLAB_COM_PAT}" https://${GITLAB_DOMAIN}/api/v4/projects/${PROJECT_ID}/packages/generic/${PACKAGE_NAME}/${PACKAGE_VERSION}/db.sql.gz -o /var/www/html/.ddev/.downloads/db.sql.gz
  service: web

files_pull_command:
  command: |
    # set -x   # You can enable bash debugging output by uncommenting
    set -eu -o pipefail
    ls /var/www/html/.ddev >/dev/null # This just refreshes stale NFS if possible
    mkdir -p /var/www/html/.ddev/.downloads
    pushd /var/www/html/.ddev/.downloads >/dev/null
    curl --header "PRIVATE-TOKEN: ${GITLAB_COM_PAT}" https://${GITLAB_DOMAIN}/api/v4/projects/${PROJECT_ID}/packages/generic/${PACKAGE_NAME}/${PACKAGE_VERSION}/files.zip -o /var/www/html/.ddev/.downloads/files.zip
  service: web

files_import_command:
  command: |
    # set -x   # You can enable bash debugging output by uncommenting
    set -eu -o pipefail
    unzip /var/www/html/.ddev/.downloads/files.zip -d ${DDEV_DOCROOT}/
    rm /var/www/html/.ddev/.downloads/files.zip
  service: web
```

```bash
ddev pull gitlab
```

----

### Example

The TYPO3 Demo project makes use of this:

- GitLab [CI Job](https://git.typo3.org/services/demo.typo3.org/site/-/blob/c7487582653edbf6e0d6420c254681e3815f141c/.gitlab-ci.yml#L131-147)
- Package Registry: [Generic Site 1.0.0](https://git.typo3.org/services/demo.typo3.org/site/-/packages/37)

---

### Filefill

Install filefill with composer:

```bash
composer require ichhabrecht/filefill
```

----

### Filefill

Copy to ext_localconf.php and change the domain:

```php
if (Environment::getContext()->isDevelopment() || Environment::getContext()->isTesting()) {
    // Configure filefill extension
    $GLOBALS['TYPO3_CONF_VARS']['EXTCONF']['filefill']['storages'][1] = [
        [
            'identifier' => 'domain',
            'configuration' => 'https://www.example.com/',
        ],
    ];

$GLOBALS['TYPO3_CONF_VARS']['LOG']['IchHabRecht']['Filefill'] = [
        'writerConfiguration' => [
            LogLevel::DEBUG => [
                FileWriter::class => [
                    'logFileInfix' => 'filefill',
                ],
            ],
        ],
    ];
}
```

---

### Hooks for comfort

- Good user experience
- Same login for all **!LOCAL!** projects

.ddev/config.yaml
```yaml
hooks:
    post-import-db:
          # Add excluded database tables
        - exec: vendor/bin/typo3cms database:updateschema
          # Clear typo3 cache
        - exec: vendor/bin/typo3cms cache:flush
          # Create local admin user
        - exec: vendor/bin/typo3cms backend:createadmin admin Password.1
```

---

## Happy pulling