wait for db with docker and python

In this post we will see how to resolve classic problem to let python application wait to mysql docker container to be ready.

We have two ways to put the waiting process:

In python application

# assume 'db' is a peewee database object

def wait_for_db_connection(max_tries, sleep_duration_in_seconds):
    try_count = 0
    while True:
        try_count += 1
        try:
            db.connect()
            logger.info("database server connection try {0}: OK".format(try_count))
            return
        except Exception as error:
            if try_count < max_tries:
                logger.info("database server connection try {0}: FAILED".format(try_count))
                time.sleep(sleep_duration_in_seconds)
                pass
            else:
                logger.error("database server connection reached max tries. Unable to connect to DB")
                logger.exception(error)
                raise

Then call wait_for_db_connection(..)

In Dockerfile

COPY ./wait-for-db.sh /wait-for-db.sh
RUN chmod +x {{ compose_project_path }}/wait-for-db.sh
CMD [ "/wait-for-db.sh", "python", "app.py" ]

Where wait-for-db.sh (jinja2 like):

#!/bin/sh
# wait-for-db.sh

set -e

host="{{ database_host }}"
port={{ database_port }}

shift
cmd="$@"

until nc -z -v -w30 $host $port; do
  >&2 echo "Database server is unavailable - sleeping"
  sleep 2
done

>&2 echo "Database server is up - executing command"
exec $cmd

Jersey backward incompatible changes in 2.26

Jersey 2.26 and newer are not backward compatible with older versions.

The reason behind that has been stated in the release notes:

Unfortunately, there was a need to make backwards incompatible changes in 2.26. Concretely jersey-proprietary reactive client API is completely gone and cannot be supported any longer - it conflicts with what was introduced in JAX-RS 2.1 (that's the price for Jersey being "spec playground..").

Another bigger change in Jersey code is attempt to make Jersey core independent of any specific injection framework. As you might now, Jersey 2.x is (was!) pretty tightly dependent on HK2, which sometimes causes issues (esp. when running on other injection containers. Jersey now defines it's own injection facade, which, when implemented properly, replaces all internal Jersey injection.

To fix errors when upgrading to 2.26 and newer, add following dependency:

<dependency>
    <groupId>org.glassfish.jersey.inject</groupId>
    <artifactId>jersey-hk2</artifactId>
    <version>2.26</version>
</dependency>

Of course, set version to 2.26 or newer if any

https://stackoverflow.com/a/46405129

renew expired let’s encrypt certificate with certbot

Firstly my certificates were created for mysubdomain.witr.net with certbot like so:

sudo apt-get install certbot -t stretch-backports
sudo certbot certonly --webroot -w /path/to/http/server/web/dir1

All is fine.

Some days after, I changed the directory of http server web path (ansible refactoring)

When certificate is near to expire, and I’m going to renew its validity, always with certbot, I’m having following error:

could not find path /path/to/http/server/web/dir1

Fixed by changing certbot configuration in /etc/letsencrypt/renewal/mysubdomain.witr.net.conf

# edit following line option to set the good path: 
mysubdomain.witr.net = /new/path

enable markdown on wordpress with wp-cli

I’m using wp-markdown plugin inside an auto generated wordpress site on docker.

Regularly, when I regenerate the site, I’m obligated to login each time to activate markdown manually.

To enable programmatically markdown features of this plugin, it’s so easy as update a worpdress option with wp-cli:

  • Inline command
wp option update markdown --format=json '{ "post_types" : { "0" : "post", "1" : "page",  "2" : "comment" }, "markdownbar" : { "0" : "posteditor", "1" : "comment" }, "prettify" : "0" }'
  • Using external file
wp option update markdown --format=json < markdown-option.txt

While markdown-option.txt contains:

{
  "post_types" :
    {
      "0" : "post",
      "1" : "page",
      "2" : "comment"
    },
  "markdownbar" :
    {
      "0" : "posteditor",
      "1" : "comment"
    },
  "prettify" : "0"
}
  • With docker exec (just skip double-quotes)
docker exec -ti --user www-data wordpress bash -c "wp option update markdown --format=json '{ \"post_types\" : { \"0\" : \"post\", \"1\" : \"page\",  \"2\" : \"comment\" }, \"markdownbar\" : { \"0\" : \"posteditor\", \"1\" : \"comment\" }, \"prettify\" : \"0\" }'"

curl in updated ubuntu Bionic does not work

curl stopped working properly after updates installed on ubuntu 18.04 with following error:

curl: /usr/local/lib/libcurl.so.4: no version information available (required by curl)
curl: (1) Protocol "https" not supported or disabled in libcurl

Check libcurl (locate) and then change symlink of usr/local libcurl to usr/lib libcurl :

sudo rm /usr/local/lib/libcurl.so.4
sudo ln -s /usr/lib/x86_64-linux-gnu/libcurl.so.4.5.0 /usr/local/lib/libcurl.so.4

This fixed the problem for me

script bash waiting for http server

How-To

Waiting for http server responding on http://localhost:8080

while [ "$(HEAD http://localhost:8080 | grep '200\ OK' | wc -l)" = "0" ]; do echo -n "."; sleep 1; done;

Explanation:

While http server is not ready, HEAD receives http code 500 with conection refused error

HEAD http://localhost:8080
500 Can't connect to localhost:8080 (Connection refused)
Content-Type: text/plain
Client-Date: Fri, 05 Apr 2019 07:11:06 GMT
Client-Warning: Internal response

When the server is ready, HEAD receives http code 200 (OK)

HEAD http://localhost:8080
200 OK
Cache-Control: no-cache, must-revalidate, max-age=0
Connection: close
Date: Fri, 05 Apr 2019 07:09:47 GMT
Server: Apache/2.4.25 (Debian)
...etc

integrate matomo in your Angular project

I wanted to share the easiest way to integrate https://github.com/matomo-org/matomo in an angular project (tested with angular 7) with zero external package.

Add following structure to angular project

src/
├── analytics
│  ├── conf
│  │  ├── matomo.conf.local.js
│  │  ├── matomo.conf.staging.js
│  │  └── matomo.conf.prod.js
│  ├── matomo.conf.js
│  └── matomo.js

Put your matomo provided script in matomo.js and :
– delimit with: if(matomoEnabled){ ... }.
– replace the matomo url with variable matomoUrl.
– replace the matomo site id by variable matomoSiteId.

if (matomoEnabled) {
  console.log("init matomo { matomoEnabled: " + matomoEnabled + ", matomoUrl: " + matomoUrl + ", matomoSiteId: " + matomoSiteId + " }");
  var _paq = window._paq || [];
  /* tracker methods like "setCustomDimension" should be called before "trackPageView" */
  _paq.push(['trackPageView']);
  _paq.push(['enableLinkTracking']);
  (function() {
    var u=matomoUrl;
    _paq.push(['setTrackerUrl', u+'matomo.php']);
    _paq.push(['setSiteId', matomoSiteId]);
    var d=document, g=d.createElement('script'), s=d.getElementsByTagName('script')[0];
    g.type='text/javascript'; g.async=true; g.defer=true; g.src=u+'matomo.js'; s.parentNode.insertBefore(g,s);
  })();
} else {
  console.log("matomo disabled");
}

In matomo.conf.js:

matomoEnabled=false;
matomoUrl="";
matomoSiteId="";

Copy matomo.conf.js content to matomo.conf.staging.js and matomo.conf.prod.js and fill with environment values.

Then import matomo.js and matomo.conf.js with angular.json:

"angular": {
  "build": {
    "options": {
      ...
      "scripts": [
        "src/analytics/matomo.conf.js",
        "src/analytics/matomo.js"
      ]
      ...
    }
  }
}

Finally use file replacements in angular.json:

"configurations": {
  "prod": {
    "fileReplacements": [
      {
        "replace": "src/analytics/matomo.conf.js",
        "with": "src/analytics/conf/matomo.conf.local.js"
      }
    ]
  }
}

ssh signing failed

When encountring following error:

ssh user@host
sign_and_send_pubkey: signing failed: agent refused operation

It’s potentially because your ssh folder hasn’t appropriate permissions.

To resolve it try:

chmod 700 ~/.ssh
chmod 600 ~/.ssh/*

docker could not resolve deb.debian.org

While building a Dockerfile which installs some packages, following error is raised:

Could not resolve 'deb.debian.org'

In fact, docker daemon can’t resolve specified host, because no dns server is set in config.

To fix issue, at least one valid dns server must be configured in /etc/docker/daemon.json like so (google dns as example):

{
    "dns": ["8.8.8.8"]
}

And then restart docker daemon sudo service docker restart

register and update docker containers ips in etc hosts

Use following script to update /etc/hosts with entries of docker containers mycontainersXXXXwitr.local

# delete lines between '# docker-compose containers start' and '# docker-compose containers end' in /etc/hosts
sudo sed -i.bak '/^# docker-compose containers start/,/# docker-compose containers end/d' /etc/hosts


echo -e "\nyour /etc/hosts is updated with following:"
echo "======================"
echo "" | sudo tee -a /etc/hosts
echo "# docker-compose containers start" | sudo tee -a /etc/hosts
echo "" | sudo tee -a /etc/hosts
docker inspect -f '{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}} {{.Name}}' $(docker ps --filter name=mycontainers.*witr.local -q) | sed 's/\///' | sudo tee -a /etc/hosts
echo "" | sudo tee -a /etc/hosts
echo "# docker-compose containers end" | sudo tee -a /etc/hosts
echo "======================"