Satellite

FEATURE STATE: Middleware 0.1 [alpha]

Satellite


A satellite is a micro-service running in the cloud, packaged as a docker image. It can be deployed in any Docker infrastructure, including Kubernetes clusters.

Building a satellite

The configuration of your satellite must be defined in a yaml file.

Setting up the Adapter

First, you should declare the docker image, or the file, on which we want to build the micro-service.

Using Docker

To use a docker image to build your micro-service, implement the docker key with its configuration options :

  • from : determines the image on which your code will run
  • workdir : define the working directory of a Docker container
  • tags : determines the references to the Docker images


satellite:
  docker:
      from: php:8.0-cli-alpine
      workdir: /var/www/html
      tags:
        - kiboko/satellite:foo
        - kiboko/satellite:bar
#...


<?php

use Kiboko\Component\ETL\Satellite\Adapter\Docker;

$dockerfile = new Docker\Dockerfile(
    new Docker\Dockerfile\From('php:8.0-cli-alpine'),
    new Docker\Dockerfile\Workdir('/var/www/html/'),
);

$satellite = (new Docker\Satellite(
    'foo/satellite:bar',
    $dockerfile,
))->addTags("kiboko/satellite:foo", "kiboko/satellite:bar");

Here, we chose to use the php:8.0-cli-alpine base image on which our code will be executed. You could use any docker image of your choice, however you will need to have a PHP runtime available, in a compatible version: >=8.0 with the CLI SAPI.

Using system

To use a system file to build your micro-service, implement the filesystem key.

The filesystem key is accompanied by a path key which determines the path of the microservice to be built.


satellite:
  filesystem:
    path: path/to/file/foo
#...



Configure composer

As a second step, we need to declare the composer dependencies our microservice will have with the composer key.

The require option allows to add all the packages, write like package_name:version, that we need for your microservice.


satellite:
#...
  composer:
    require:
      - "php-etl/pipeline:^0.2"
      - "php-etl/fast-map:^0.2"
      - "php-etl/csv-flow:^0.1"
      - "akeneo/api-php-client-ee"
      - "laminas/laminas-diactoros"
      - "php-http/guzzle7-adapter"


$dockerfile->push(
    new Docker\PHP\Composer(),
    new Docker\PHP\ComposerInit(),
    new Docker\PHP\ComposerMinimumStability('dev'),
    new Docker\PHP\ComposerRequire('php-etl/pipeline:@dev')
    new Docker\PHP\ComposerRequire('php-etl/fast-map:@dev');
);

The autoload option is optional and allows you to configure your autoloader by specifying one or more namespaces and and directories paths as if you were directly in the composer.json.

satellite:
#...
  composer:
    autoload:
      psr4:
        - namespace: "Pipeline\\"
          paths: [""]

The from_local option is optional and copies local repositories instead of downloading them.

satellite:
#...
  composer:
    from_local: true

Setting up the runtime

Now that we have made our environment prepared for our satellite, we will declare the way we want our pipeline to handle our data flows.

There are 4 types of runtimes, depending on your needs you will have to choose one of:

  • http-api: the micro-service will be operating an API, on which several URL routes can be registered. http-api is used for REST API.
  • http-hook: the micro-service will be operating an API on a single URL route. http-hook is used for webhooks. A webhook is a POST request sent to a URL. It’s considered to be a means for one application to provide other applications with real-time information
  • pipeline: the micro-service will be operating a data pipeline, executed in the backend that can be executed as a cron job. For more information on pipelines, see Pipeline.
  • workflow: the micro-service will be orchestrating more than one data pipeline, executed in the backend that can be executed as a cron job

Using Pipeline

Please visit the Pipeline documentation page to find out how to set up your pipeline.


satellite:
   # ...
   pipeline:
      steps:
      - akeneo:
          enterprise: true
          extractor:
            type: productModel
            method: all
            search:
              - { field: enabled, operator: '=', value: true }
              - { field: completeness, operator: '>', value: 70, scope: ecommerce }
              - { field: completeness, operator: '<', value: 85, scope: ecommerce }
              - { field: categories, operator: IN, value: winter_collection }
              - { field: family, operator: IN, value: [camcorders, digital_cameras] }
          logger:
            type: 'stderr'
      - fastmap:
          map:
            - field: '[sku]'
              copy: '[sku]'
            - field: '[title]'
              expression: 'input["sku"] ~" | "~ input["name"]'
            - field: '[name]'
              copy: '[name]'
            - field: '[staticValue]'
              constant: 'Lorem ipsum dolor sit amet, consectetur adipiscing elit. Curabitur mollis efficitur justo, id facilisis elit venenatis et. Sed fermentum posuere convallis. Phasellus lectus neque, bibendum sit amet enim imperdiet, dignissim blandit nisi. Donec nec neque nisi. Vivamus luctus facilisis nibh id rhoncus. Vestibulum eget facilisis tortor. Etiam at cursus enim, vitae mollis ex. Proin at velit at erat bibendum ultricies. Duis ut velit malesuada, placerat nisl a, ultrices tortor.'
            - field: '[foo]'
              expression: 'input'
              list:
                - field: '[bar]'
                  copy: '[bar]'
      - csv:
          loader:
            file_path: output.csv
            delimiter: ','
            enclosure: '"'
            escape: '\'


<?php

/** @var array $config */ 
$pipeline = new Runtime\Pipeline($config);
$pipeline->build();

Using Workflow

Please visit the Workflow documentation page to find out how to set up your workflow.

satellite:
  # ...
  workflow:
    jobs:
    - name: 'Lorem ipsum dolor'
      pipeline:
        steps:
        - akeneo:
            extractor:
              type: category
              method: all
            client:
              api_url: 'http://localhost:8080'
              client_id: '5_5dxapwm2rs4kkso8wgwk08o88gg0wc0owkgsgg0gkcwos4o0wo'
              secret: 34n14k1ajy800g0ocww8w8cwckogoc844ggwcs8gkg8w4k4888
              username: 'kiboko_2145'
              password: 62d0f1090
        - csv:
            loader:
              file_path: categories.csv
              delimiter: ','
              enclosure: '"'
              escape: '\\'
    - pipeline:
        steps:
        - akeneo:
            extractor:
              type: product
              method: all
            client:
              api_url: 'http://localhost:8080'
              client_id: '5_5dxapwm2rs4kkso8wgwk08o88gg0wc0owkgsgg0gkcwos4o0wo'
              secret: 34n14k1ajy800g0ocww8w8cwckogoc844ggwcs8gkg8w4k4888
              username: 'kiboko_2145'
              password: 62d0f1090
        - csv:
            loader:
              file_path: products.csv
              delimiter: ','
              enclosure: '"'
              escape: '\'

Configuration formats

There are 2 ways to declare satellites :

Execute your micro-service

After configuring your config file, you can run the command which will allow you to create the Dockerfile or the file system.

# will execute the satellite.yaml file located at the root of the project
php bin/console build

# specify the name of the file to be executed
php bin/console build `satellite.yaml`

This command will create a folder with a file function.php containing the pipeline code.

You have to execute it with php command like this :

php path/to/folder/function.php