Chuan Chuan Law

DevOps | Software Automation | Continuous Integration

Ansible – How to fix Failed to import docker-py – No module named websocket. Try `pip install docker-py`

Set the path of Python site-packages in your playbook:

environment:
PYTHONPATH: “{{ lookup(‘env’,’PYTHONPATH’) }}:/Users/claw/Library/Python/2.7/lib/python/site-packages:/Users/claw/Library/Python/2.7/lib/python/site-packages”

How To Migrate Elastic Search Cluster To New Cluster

Introduction

There are times when we will choose to set up a new cluster and migrate the data instead of updating existing one especially when when the version that we want to upgrade is many versions ahead of the current one. Thus, the risk associated with the upgrade will be higher and downtime of existing cluster due to the upgrade will also be longer.

Setup The New Elk Cluster

Install the new versions in the following sequence:

  • Elasticsearch
  • Kibana
  • Logstash

The Elastic Search official guide contains comprehensive guide on the installation, so this blog will talk about personal experience and problems encountered which are not part of the installation guide.

Elastic Search

The main thing here is to do Snapshot and Restore of current Elastic Search cluster to the new one.

  • Register a backup with the current ES. In this blog, we use S3 repository

curl -X PUT “http://oldelasticsearch:9200/_snapshot/s3_repository?verify=false” -H ‘Content-Type: application/json’ -d’
{
“type”: “s3”,
“settings”: {
“bucket”: “es-snapshot”,
“region”: “us-east-1”
}
}

  • Snapshot to S3

curl -XPUT “http://oldelasticsearch:9200/_snapshot/s3_repository/snap1?pretty?wait_for_completion=true

  • Register the backup on the new cluster

curl -X PUT “http://newelasticsearch:9200/_snapshot/s3_repository?verify=false” -H ‘Content-Type: application/json’ -d’
{
“type”: “s3”,
“settings”: {
“bucket”: “es-snapshot”,
“region”: “us-east-1”
}
}

  • Restore from S3 bucket on the new ES cluster

curl -X POST “http://newelasticsearch:9200/_snapshot/s3_repository/snap1/_restore

Kibana

The hip cups that you will see from the newly installed Kibana will be due to some conflicting indices from new and old cluster.

To solve this, do the following:

  • Close the kibana index

curl -X POST “http://newelasticsearch:9200/.kibana/_close

  • Delete the kibana index

curl -X DELETE “http://newelasticsearch:9200/.kibana

  • Restore kibana index from S3 (old cluster)

curl -X POST “http://newelasticsearch:9200/_snapshot/s3_repository/snap1/_restore” -H ‘Content-Type: application/json’ -d’
{
“indices”: “.kibana”,
“ignore_unavailable”: true,
“include_global_state”: true
}

  • Open the kibana index again

curl -X POST “http://newelasticsearch:9200/.kibana/_open

  • Restart kibana

Another issue with Kibana setup will be the Logtrial plugin. The plugin version needs to match exactly the Kibana version, thus we will need to do some manual hacks.

Below are the hacks in Ansible script:

  • Download logtrial

– name: download logtrail
get_url:
url: https://github.com/sivasamyk/logtrail/releases/download/v0.1.23/logtrail-5.6.5-0.1.23.zip
dest: /tmp

  • Unzip logtrial

– name: unzip logtrail
unarchive:
src: /tmp/logtrail-5.6.5-0.1.23.zip
dest: /tmp

  • Modify the kibana version in package.json

– name: modify the kibana version in package.json
lineinfile:
path: /tmp/kibana/logtrail/package.json
regexp: ‘”version”: “5.6.5”‘
line: ‘”version”: “5.6.15”‘

  • Zip it back

– name: zip logtrial back
archive:
path: /tmp/kibana
dest: /usr/share/kibana/bin/logtrail-5.6.5-0.1.23.zip
format: zip
mode: 0664

  • Install the modified logrial

– name: install modified logtrial
shell: ./kibana-plugin install file:///usr/share/kibana/bin/logtrail- 5.6.5-0.1.23.zip
chdir=/usr/share/kibana/bin

How To Solve DNS Issues in Ubuntu 18.04

Introduction

Ubuntu 18.04 Bionic has DNS bug where it is not able to resolved DNS.

Solution

We can do the following hack to solve it:

sudo rm -f /etc/resolv.conf
sudo ln -s /run/systemd/resolve/resolv.conf /etc/resolv.conf
reboot

Jenkins – How To Manually Upgrade Jenkins Plugin To Specific Versions

Introduction

Jenkins Plugin Manager will enable plugin installation of the latest version. Sometimes we might want to install plugin of a specific version, instead of the latest. To do that, we can do the following:

Steps

  • Older versions of plugins can be found here: https://updates.jenkins.io/download/plugins/
  • The plugins are in .hpi file extensions
  • Download the version you want and put it on the Jenkins server
  • Plugins are stored in /var/lib/jenkins
  • Backup – you might want to backup the current version of the plugins before upgrading. Each plugin has a directory and a .jpi file.

For eg: datadog plugin

mkdir datadog_backup

mv datadog* /datadog_backup

  • Place the .hpi file in /var/lib/jenkins
  • Restart Jenkins

service jenkins restart

  • A directory with the plugin name and a .jpi file will be created upon restart
  • Check if the plugin with the correct version appears on Jenkins website Manage Jenkins->Plugin Manager
  • If not, check the log on the server in /var/log/jenkins 
  • Log is usually called jenkins.log
  • Log will show the plugin installation failure as SEVERE
  • Usually this is due to version dependencies of other plugins as named in the log.

For eg:

SEVERE: Failed Loading plugin sauce-ondemand
java.io.IOException: Dependency workflow-job (1.15), workflow-cps (1.15), workflow-basic-steps (1.15), workflow-step-api (1.15) doesn’t exist

  • Download and install the missing dependencies plugin via the same method as above
  • Upon successful installation, you will see the plugin with the right version appearing in Manage Jenkins->Plugin Manager

Ubuntu 18 – How To A Remove Configure Grub-Pc Pop Up During Apt-Get Update

  • The above pop up will appear waiting for user interaction during apt-get -y upgrade
  • Add the following to before the apt-get upgrade command

export DEBIAN_FRONTEND=noninteractive

How To Build & Compile Source Into Debian Package

  • Install dh-make

sudo apt-get install build-essential dh-make

  • Put your source code in tar.gz into a directory. File format is in <filename>-<version>

mkdir mysource

mv mysource-01.tar.gz mysource/

cd mysource

tar -xvzf mysource-01.tar.gz

  • Create the initial Debian package

cd mysource-01

dh_make -f ../mysource-01.tar.gz

  • It will prompt to ask the type of package. Then hit enter to confirm.

Type of package: single binary, indep binary, multiple binary, library, kernel module, kernel patch?
[s/i/m/l/k/n] s

  • A debian folder will be created under mysource-01 with all the necessary debian files

total 108
drwxr-xr-x 3 root root 4096 Dec 6 11:39 .
drwxr-xr-x 3 root root 4096 Dec 6 11:39 ..
-rw-r–r– 1 root root 190 Dec 6 11:39 changelog
-rw-r–r– 1 root root 2 Dec 6 11:39 compat
-rw-r–r– 1 root root 525 Dec 6 11:39 control
-rw-r–r– 1 root root 1679 Dec 6 11:39 copyright
-rw-r–r– 1 root root 0 Dec 6 11:39 docs
-rw-r–r– 1 root root 4596 Dec 6 11:39 init.d.ex
-rw-r–r– 1 root root 1646 Dec 6 11:39 manpage.1.ex
-rw-r–r– 1 root root 4663 Dec 6 11:39 manpage.sgml.ex
-rw-r–r– 1 root root 11018 Dec 6 11:39 manpage.xml.ex
-rw-r–r– 1 root root 129 Dec 6 11:39 menu.ex
-rw-r–r– 1 root root 134 Dec 6 11:39 mysource.cron.d.ex
-rw-r–r– 1 root root 238 Dec 6 11:39 mysource.default.ex
-rw-r–r– 1 root root 526 Dec 6 11:39 mysource.doc-base.EX
-rw-r–r– 1 root root 958 Dec 6 11:39 postinst.ex
-rw-r–r– 1 root root 931 Dec 6 11:39 postrm.ex
-rw-r–r– 1 root root 691 Dec 6 11:39 preinst.ex
-rw-r–r– 1 root root 878 Dec 6 11:39 prerm.ex
-rw-r–r– 1 root root 186 Dec 6 11:39 README.Debian
-rw-r–r– 1 root root 271 Dec 6 11:39 README.source
-rwxr-xr-x 1 root root 114 Dec 6 11:39 rules
drwxr-xr-x 2 root root 4096 Dec 6 11:39 source
-rw-r–r– 1 root root 787 Dec 6 11:39 watch.ex

  • You can then modify the relevant files for eg: control

Python: ImportError: /usr/local/lib/python2.7/lib-dynload/_io.so: undefined symbol: PyUnicodeUCS2_Replace

The above error might occur if you have compiled and installed Python from source on Ubuntu.

curl https://www.python.org/ftp/python/2.7.6/Python-2.7.6.tgz -o python.tgz

tar -xvzf python.tgz

cd Python-2.7.6/

./configure

make

make install

 

The error can be fixed by

./configure –enable-unicode=ucs4

Docker – How To Set Locale In Ubuntu

Add the following in your Dockerfile to set Locale in Ubuntu

RUN locale-gen en_US.UTF-8
ENV LANG en_US.UTF-8
ENV LANGUAGE en_US:en
ENV LC_ALL en_US.UTF-8

Jenkins – config.xml

If there is a need to debug Jenkins configuration and the when the interface set up looks correct, it’s better to find more detailed configuration information via the config.xml.

  1. In the server where Jenkins is installed, usually in path /var/lob/jenkins/jobs
  2. Browse to your job folder
  3. There will be a config.xml which will contain the job configuration details

Jenkins – How To Disable Jobs

How To Disable One Jenkins Job

  • Get breadcrumb

curl -u “<user>:<password>“ ‘https://<jenkins_url>/crumbIssuer/api/xml?xpath=concat(//crumbRequestField,”:”,//crumb)’

 

  • curl -I -X POST https://<jenkins_url>/<job_path>/disable –user <user>:<password> -H “<jenkins bread crumb>”

How To Disable Jenkins jobs Under A Folder

  • Via Jenkins script

https://<jenkins_url>/script

  • Enter the following in the script and execute it

folderName=”Build/APP/test” //full name of the folder you want to disable all jobs in

Jenkins.instance.getItemByFullName(folderName).allJobs

.each {

      it.setDisabled(true)

      println(“Disabled job: [$it.fullName]”)

    }

null

Note: Look for file structure in server, ignore the “jobs” path in URL

« Older posts

© 2019 Chuan Chuan Law

Theme by Anders NorenUp ↑