🪄Installation

A complete guide to install Episilia

Prerequisites

AWS, GCP, Digital Ocean are supported out of the box. For installation in a different cloud provider or in your own datacenter, any S3 compatible Block Storage can be supplied, such as minio server.
Helm version 3.0+ (Helm CLI)
Kubectl CLI
A running Kubernetes cluster. Minimum of 2 cores and 4 GB memory required
Kafka for sourcing logs. Other topics to be created during installation
S3 credentials and a bucket to store index and data files
AVX2 support needed on compute machines

On linux (or unix machines) information about your cpu is in /proc/cpuinfo. You can extract information from there by hand, or with a grep command (grep flags /proc/cpuinfo). Also most compilers will automatically define AVX2 so you can check for that too

Using Helm

Step 1 : Add Helm Repo

Repo URL: https://episilia.gitlab.io/episilia-helm/release

helm repo add <NAME> <URL>
$ helm repo add episilia https://episilia.gitlab.io/episilia-helm/release

Listing chart repositories:

$ helm repo list or helm repo ls
  NAME        URL 
  episilia   https://episilia.gitlab.io/episilia-helm/release

Searching for charts in the repository:

$ helm search repo episilia

Step 2 : Configure values

Update global values in the episilia/episilia-cpanel master chart values.yaml file. All configurable values are explained in the section below titled "Configuration".

Inspect the values before installing application use below:

$ helm inspect values episilia/episilia-cpanel > episilia_values.yaml

$ helm install episilia episilia/episilia-cpanel --set global.client.id=episilia-client --set global.client.env=dev --set global.client.license.key=episilia

Step 3 : Perform dry run

helm install  episilia episilia/episilia-cpanel -f episilia_values.yaml --dry-run

$ helm install episilia episilia/episilia-cpanel --set global.client.id=episilia-client --set global.client.env=dev --dry-run --set global.client.license.key=episilia

Step 4 : Install Helm Repo

helm install <RELEASE NAME> <CHART>
$ helm install episilia episilia/episilia-cpanel  -f episilia_values.yaml
                              or      
$ helm install episilia episilia/episilia-cpanel --set global.client.id=episilia-client --set global.client.env=dev --set global.client.license.key=episilia

Listing installed helm charts

 $ helm ls

Listing pods

 $ kubectl get pods

List services:

 $ kubectl get services

Step 5 : Access Grafana Logs Browser

Access Grafana Logs Browser using grafana cluster IP. Redirect Grafana IP as needed.

Default USERNAME and PASSWORD is admin

Configuration

Episilia can be configured with the following values

Image tag

Docker image for episilia.

imageTag: &release "1.1.4"

Enabling server nodes

Enable required servers.

episilia-log-indexer, episilia-search, episilia-gateway needs to be enabled by default.

Disable grafana if using your own Grafana instance

episilia-log-indexer:
  enabled: true
episilia-search:
  enabled: true
episilia-search-fixed:
  enabled: false
episilia-gateway:
  enabled: true
episilia-log-indexer-opt:
  enabled: false
episilia-alert:
  enabled: false
grafana:
  enabled: true

License and Client id

License key and Client id can be obtained from https://crm.episilia.com

env denotes the client identified environment. Examples include dev / qa / prod

global:
  client:
    id: episilia-helm
    env: test-helm
    license:
      key: episilia

All common ops

Common ops config goes below.

ops:
    log:
      debug: on # Enable to get debug logs in all the servers
    license:
      url: episilia-cpanel:8080  
    cpanel:
      data:
        publish:
          interval:
            seconds: 300 # Time interval in which cpanel will be pushing metrics to console

Kafka config

All kafka specific configuration goes below.

kafka:
    metadata:
      broker:
        list: localhost:9092 # The kafka broker
    group:      
      search: episilia-search-group   # Kafka consumer-group for search
      cpanel: episilia-cpanel-group   # Kafka consumer-group for cpanel
      logwatcher: episilia-logwatcher-group  # Kafka consumer-group for log-watcher
      logwatcher.tail: episilia-logwatcher-tail-group  # Kafka consumer-group for log-watcher-tail
      logwatcher.alert: episilia-logwatcher-alert-group  # Kafka consumer-group for log-watcher-alert
    topic:
      index:
        live: stagefiles-topic # Topic for internal publish indexed files - stage.topic
        optimized: optfiles-topic # Topic for internal optimize.topic:  publish file names post optimization
      optimize:
        request: stagefolder-topic #optimize.request.topic send folders to optimize
      cpanel:
        in: cpaneld-topic # Internal topic cpanel.data.topic
        out: cpaneld-topic # Internal topic cpanel.data.topic
      alert:
        response:
          out: alert_out 
    indexer:
      logs:
        topics: logs # Topic from where logs are loaded.
      group: episilia-indexer-group  # Kafka consumer-group for indexer

Datastore

S3 bucket and folder details goes below.

datastore:
    s3:
      accesskey: ""  # filestorage access key (eg: AWS S3 access key)
      secretkey: ""  # filestorage secret key (eg: AWS S3 secret key)
      region: ""     # filestorage region (eg: AWS S3 region)
      endpoint:
        url: ""     # filestorage endpoint URL (eg: AWS S3 endpoint URL)
      sign:
        payload: true
      bucket: episilia-bucket  # filestorage bucket (eg: AWS S3 bucket)
      folder: episilia-folder  # filestorage folder URL (eg: AWS S3 folder)
      work:
        folder: work-folder   # Folder name to store s3 limit/write results
      url:
        prefix: s3://   # filestorage prefix URL (default to AWS S3)

Indexer

Config for indexer and optimizer.

indexer:
    image:
      repository: episilia/episilia-log-indexer    # docker image of episilia-log-indexer
      tag: *release
    replicaCount: "1"        # kubernetes pod replicas of episilia-log-indexer
    resources:
      limits:
        cpu: 800m            # cpu limit on episilia-log-indexer 
        memory: 1024Mi       # memory limit on episilia-log-indexer
      requests:
        cpu: 400m            # cpu request on episilia-log-indexer
        memory: 300Mi        # memory request on episilia-log-indexer


    logs:
      source: kafka         # source: kafka  # s3 or kafka
    schema:
      appid:
        fixed: "defaultApp"      # If appid is a fixed string
        keys: "project.app_id"   # label(s) for app identifier
      tenantid:        
        fixed: "defaultTenant"   # If tenantid is a fixed string  
        keys: ""                 # label(s) for tenant identifier        
      message:
        key: "log"               # actual log message key
      timestamp:
        key: "time"              # timestamp key
        formats: "%Y-%m-%dT%H:%M:%S"  #to specify timestamp format (ex: %Y-%m-%dT%H:%M:%S )
      exclude: "time"            # labels to be excluded from the list

  optimizer:
    replicaCount: "1"           # kubernetes pod replicas of episilia-optimizer  
    resources:
      limits:
        cpu: 800m              # cpu limit on episilia-optimizer
        memory: 1024Mi         # memory limit on episilia-optimizer
      requests:
        cpu: 500m              # cpu request on episilia-optimizer        
        memory: 300Mi          # memory request on episilia-optimizer

Alert

Config for alert server .

  alert:
    resources:
      limits:
        cpu: 500m              # cpu limit on episilia-optimizer
        memory: 600Mi         # memory limit on episilia-optimizer
      requests:
        cpu: 300m              # cpu request on episilia-optimizer
        memory: 300Mi          # memory request on episilia-optimizer
    rules.file.url:            #s3 path of alert rules file
    prometheus.gateway: localhost:5070  # push-gateway url

Search

Config for search server goes below.

search:
    image:
      repository: episilia/episilia-search   # docker image of episilia-search
      tag: *release
    replicaCount: "1"                        # kubernetes pod replicas of episilia-search 
    resources:
      limits:
        cpu: "1"                             # cpu limit on episilia-search
        memory: 2048Mi                       # memory limit on episilia-search
      requests:
        cpu: 500m                            # cpu request on episilia-search
        memory: 600Mi                        # memory request on episilia-search

    api:
      timeout:
        seconds: 40                         # timeout for search while querying
    
    live:
      from:
        hours: 48                           # hours from when the required index blocks should be loaded
      to:
        hours: 0                            # hours till when the required index blocks should be loaded, Note: value to be "0" to get instant logs
    labels:
      exclude: "@timestamp,log" # Lables excluded from grafana dropdown GUI.

Config for historic search server.

fixedSearch:
    bucket: ""             # s3 bucket for historic search to run parallelly, Note: if the value is empty it takes datastore.s3.bucket value as default
    folder: ""             # s3 folder for historic search to run parallelly, Note: if the value is empty it takes datastore.s3.folder value as default
    replicaCount: "1"      # kubernetes pod replicas of historic episilia-search 
    resources:
      limits:
        cpu: "1"          # cpu limit on historic episilia-search
        memory: 1024Mi    # memory limit on historic episilia-search
      requests:
        cpu: 500m         # cpu request on historic episilia-search
        memory: 600Mi     # memory request on historic episilia-search
    fixed:
      from:
        yyyymmddhh: "2021092100"    # the date from when the required index blocks should be loaded
      to:
        yyyymmddhh: "2021092202"    # the date till when the required index blocks should be loaded

    api:
      timeout:
        seconds: 40       # timeout for search while querying

    labels:
      exclude: "@timestamp,log" # Lables excluded from grafana dropdown GUI.

Gateway

Gateway specific configuration goes below.

  gateway:
    image:
      repository: episilia/episilia-gateway   # docker image of episilia-gateway
      tag: *release
    service:
      type: ClusterIP                         # service type of episilia-gateway
    replicaCount: "1"                         # kubernetes pod replicas of episilia-gateway
    resources:
      limits:
        cpu: 500m                             # cpu limit on episilia-gateway
        memory: 600Mi                         # memory limit on episilia-gateway
      requests:
        cpu: 300m                             # cpu request on episilia-gateway
        memory: 200Mi                         # memory request on episilia-gateway

    search:
      timeout:
        seconds: 40                           # timeout of the query from gateway

Control Panel

Control Panel specific configuration goes below.

  cpanel:
    ops:
      healthchecks:
        interval:
          mins: "5"                          # time interval in which metrics are pushed
        exclude:
          list: ""                           # to exclude the specific metrics
    api:
      access:                                # Unique keys to access the episilia console
        key: token                           
        token: random
      post:
        server: "https://console.episilia.com/publish_cpanel_data"      # Url of the console
      get:
        server: ""
      grpc:
        port: "8080"
      metrics:
        port: "8090"

Persistence Volume

If PV is enabled, configure the same below.

  persistence:
    enabled: false                              # to enable PVC 
    mountPath: "/data"                          # mount path of PVC 
    storageClassName: do-block-storage          # storage class name (differs on the cloud services that are used)
    accessModes:
    - ReadWriteOnce                             # access modes
    size: "40Gi"                                # size of PVC which will be mounted to episilia-search for live search
    historicSize: "20Gi"                        # size of PVC which will be mounted to episilia-search for historic search
    # annotations: {}
    finalizers:
     - kubernetes.io/pvc-protection
    # selectorLabels: {}
    # subPath: ""
    # existingClaim:

References

External Resources

Kubectl CLI kubectl

Helm3 CLI helm3

Upload a chart to Kubernetes

helm install <RELEASE NAME> <CHART>

helm install <CHART> —generate-name 

helm install <NAME> <CHART> —dry-run --debug

Remove a chart repository:

helm repo remove | rm <NAME>
helm repo remove episilia

To view template of the chart, before installation

helm  template <CHART>
helm template episilia/episilia-cpanel

Passing keywords at runtime during installation

helm install episilia episilia/episilia-log-indexer --set image.repository=<imagename> --set image.tag=<tagname>

Upgrade the chart with a specific release

helm upgrade <RELEASE> <CHART> 
helm upgrade episilia episilia/episilia-cpanel

flag -i or –install can be specified to run an install before if a release by this name doesn’t already exist. Otherwise, perform a rollback. If revision is not specified, the chart is rolled back to the previous version.

helm rollback <RELEASE> <REVISION>
helm rollback episilia --revision 1

View all historical revisions for a given release

helm history <RELEASE>

Uninstall a release

helm uninstall <RELEASE>
helm uninstall episilia

PreviousOverview NextSearching Logs

Last updated 10 days ago