Exporting a snapshot to S3

HBase snapshots can be exported with the command hbase snapshot export. To be able to export to S3, the AWS libraries from Hadoop must be on the classpath. These libraries are contained in the HBase image at /stackable/hadoop/share/hadoop/tools/lib/. The script export-snapshot-to-s3 facilitates the export, by providing the associated classpath, extending the Hadoop configuration with the S3 configuration, and calling hbase snapshot export. The S3 configuration is read from environment variables. The script can be directly called on the HBase master container:

$ export-snapshot-to-s3  --help
Options:
    --snapshot <arg>       Snapshot to restore.
    --copy-to <arg>        Remote destination hdfs://
    --copy-from <arg>      Input folder hdfs:// (default hbase.rootdir)
    --target <arg>         Target name for the snapshot.
    --no-checksum-verify   Do not verify checksum, use name+length only.
    --no-target-verify     Do not verify the integrity of the exported snapshot.
    --no-source-verify     Do not verify the source of the snapshot.
    --overwrite            Rewrite the snapshot manifest if already exists.
    --chuser <arg>         Change the owner of the files to the specified one.
    --chgroup <arg>        Change the group of the files to the specified one.
    --chmod <arg>          Change the permission of the files to the specified one.
    --mappers <arg>        Number of mappers to use during the copy (mapreduce.job.maps).
    --bandwidth <arg>      Limit bandwidth to this value in MB/second.

$ export \
    AWS_ACCESS_KEY_ID=myS3AccessKeyId \
    AWS_SECRET_KEY=myS3SecretKey \
    AWS_ENDPOINT=https://s3endpoint:9000/ \
    AWS_SSL_ENABLED=true \
    AWS_PATH_STYLE_ACCESS=true
$ export-snapshot-to-s3 \
    --snapshot my-snapshot \
    --copy-to s3a://my-bucket/my-snapshot

Snapshots can also be imported from S3 into HDFS:

$ export-snapshot-to-s3 \
    --snapshot snap \
    --copy-from s3a://my-bucket/my-snapshot \
    --copy-to hdfs://simple-hdfs/hbase

However, the better approach is to create a Job:

---
apiVersion: batch/v1
kind: Job
metadata:
  name: export-hbase-snapshot
spec:
  template:
    spec:
      containers:
      - name: hbase
        image: docker.stackable.tech/stackable/hbase:2.4.17-stackable0.0.0-dev
        volumeMounts:
        - name: hbase-config
          mountPath: /stackable/conf
        env:
        - name: HBASE_CONF_DIR
          value: /stackable/conf
        - name: HADOOP_CONF_DIR
          value: /stackable/conf
        - name: AWS_ENDPOINT
          value: https://s3endpoint:9000/
        - name: AWS_SSL_ENABLED
          value: true
        - name: AWS_PATH_STYLE_ACCESS
          value: true
        - name: AWS_ACCESS_KEY_ID
          valueFrom:
            secretKeyRef:
              name: s3-credentials
              key: access-key-id
        - name: AWS_SECRET_KEY
          valueFrom:
            secretKeyRef:
              name: s3-credentials
              key: secret-key
        command:
        - export-snapshot-to-s3
        args:
        - --snapshot
        - my-snapshot
        - --copy-to
        - s3a://hbase/my-snapshot
      volumes:
      - name: hbase-config
        projected:
          sources:
          - configMap:
              name: simple-hdfs
          - configMap:
              name: simple-hbase-master-default
      restartPolicy: Never
---
apiVersion: v1
kind: Secret
metadata:
  name: s3-credentials
data:
  access-key-id: myS3AccessKeyId
  secret-key: myS3SecretKey