Running NiFi in a Docker Container

Summary

I've written a few articles on Apache NiFi and how to install it on Ubuntu. I recently ran into a situation where it wasn't possible to install NiFi directly, but it could be run via a docker container. Fortunately there are Apache NiFi docker images to accomplish this. We'll be using docker-compose in this example.

Setup

The first assumption is that you already have docker up and running, and ports 514/tcp, 514/udp, 8080/tcp, and 8443/tcp are available. You can always change these in the docker-compose.yml file too. We'll also create a few volumes that can be used for persistence: the cert directory for certificates, and the drivers directory for things like MaxMind or JDBC drivers. This will get created automatically once we bring up the container.

sudo mkdir -p /opt/nifi
sudo chown -R 1000:1000 /opt/nifi

The next step is to create the docker-compose file. First do the following:

sudo mkdir -p /opt/dockerfile/nifi
sudo vi /opt/dockerfiles/nifi/docker-compose.yml

And then add the following to the above file:

version: '3'

services:
  nifi:
    cap_add:
      - NET_ADMIN # low port bindings
    image: apache/nifi
    container_name: nifi
    ports:
      - "8080:8080/tcp" # HTTP interface
      - "8443:8443/tcp" # HTTPS interface
      - "514:514/tcp" # Syslog
      - "514:514/udp" # Syslog
      - "2055:2055/udp" # NetFlow
    volumes:
      - /opt/nifi/drivers:/opt/nifi/nifi-current/drivers
      - /opt/nifi/certs:/opt/certs
      - nifi-conf:/opt/nifi/nifi-current/conf
    restart: unless-stopped

volumes:
  nifi-conf:

Next we'll bring up the containers with the following command:

cd /opt/dockerfiles/nifi/
sudo docker-compose up -d

# If you want to watch the output.
sudo docker logs -f nifi

Now that it's up, we'll tweak a few configuration options to add more RAM:

# Enter the docker container as root.
sudo docker exec -u 0 -it nifi bash

# At this point we're in the docker container.
# Install vim since nobody seems to think a text editor was necessary.
apt-get install vim

# Edit the NiFi bootstrap configuration.
vi /opt/nifi/nifi-current/conf/bootstrap.conf

# Edit out the following lines and change them.
# JVM memory settings
#java.arg.2=-Xms512m
#java.arg.3=-Xmx512m
java.arg.2=-Xms2g
java.arg.3=-Xmx2g

# Exit the docker container.
exit

# At this point we're back on the host.
# Restart the docker container.
sudo docker restart nifi

Conclusion

...and that's it. Once finished, you'll be able to access NiFi on port 8080 on the hosts, e.g. http://nifi.domain.local:8080/nifi.

Bonus! SSL and Authentication for NiFi

Disclaimer

This isn't the best security, but it's better than nothing. Configuring NiFi for authentication isn't the easiest process in the world. In this example we'll be using an NGINX docker container to act as an SSL front-end for unencrypted NiFi, and add basic user authentication on top of that.

Setup

We'll first create the directories:

sudo mkdir -p /opt/nginx/conf /opt/nginx/keys /opt/nifi/drivers /opt/dockerfiles/nifi/

We'll then create a self-signed SSL certificate for NGINX:

cd /opt/nginx/keys
sudo openssl req -x509 -sha256 -newkey rsa:2048 -keyout certificate.key -out certificate.crt -days 1024 -nodes -subj '/CN=my.domain.internal'

You'll end up with two files: certificate.key and certificate.crt. We'll be using these in the NGINX configuration to enable SSL. Next, we'll create the NGINX configuration file.

sudo vi /opt/nginx/conf/nginx.conf

Add the following to the file above:

events {
}

http {
  server {
    listen 8443 ssl;
    server_name my.domain.internal;
    ssl_certificate /keys/certificate.crt;
    ssl_certificate_key /keys/certificate.key;
    proxy_ssl_certificate /keys/certificate.crt;
    proxy_ssl_trusted_certificate /keys/certificate.key;

    location /nifi {
      proxy_pass http://nifi:8080;
      proxy_set_header Host $http_host;
      proxy_set_header X-ProxyScheme https;
      auth_basic "NiFi Authentication";
      auth_basic_user_file /keys/nginx.htpasswd;
    }
  }
}

We'll now use a modified docker-compose.yml from the original steps.

sudo vi /opt/dockerfiles/nifi/docker-compose.yml

Add the following to the file above:

version: '3'

services:
  nifi:
    cap_add:
      - NET_ADMIN # low port bindings
    image: apache/nifi
    container_name: nifi
    ports:
      - "514:514/tcp" # Syslog
      - "514:514/udp" # Syslog
      - "2055:2055/udp" # NetFlow
    volumes:
      - /opt/nifi/drivers:/opt/drivers
      - nifi-conf:/opt/nifi/nifi-current/conf
    restart: unless-stopped

  nginx:
    image: nginx
    container_name: nginx
    volumes:
      - /opt/nginx/conf/nginx.conf:/etc/nginx/nginx.conf
      - /opt/nginx/keys:/keys
    ports:
      - "8443:8443/tcp" # HTTPS interface
    restart: unless-stopped

volumes:
  nifi-conf:

Next we'll create a password for user authentication. You'll need htpasswd installed, or you can find a site that will generate one for you. Please note this is using md5 for the encryption since bcrypt doesn't always seem to play nice.

htpasswd -n admin > /opt/nginx/keys/nginx.htpasswd

# Checking the file for output.
cat /opt/nginx/keys/nginx.htpasswd
admin:$apr1$cUjKlTPI$KlnGph95k4OEBGSA3MSac0

Finally, let's bring up the docker containers:

cd /opt/dockerfiles/nifi
docker-compose up -d

Conclusion

At this point you should be prompted to log in when attempting to access NiFi. Please note that your URL will look something like this:

https://nifi.domain.local:8443/nifi

You'll need to use the port specified above and nifi at the end of the URL. Hope the helps!