Chuan Chuan Law

DevOps | Software Automation | Continuous Integration

Year: 2018 (page 1 of 2)

How To Build & Compile Source Into Debian Package

  • Install dh-make

sudo apt-get install build-essential dh-make

  • Put your source code in tar.gz into a directory. File format is in <filename>-<version>

mkdir mysource

mv mysource-01.tar.gz mysource/

cd mysource

tar -xvzf mysource-01.tar.gz

  • Create the initial Debian package

cd mysource-01

dh_make -f ../mysource-01.tar.gz

  • It will prompt to ask the type of package. Then hit enter to confirm.

Type of package: single binary, indep binary, multiple binary, library, kernel module, kernel patch?
[s/i/m/l/k/n] s

  • A debian folder will be created under mysource-01 with all the necessary debian files

total 108
drwxr-xr-x 3 root root 4096 Dec 6 11:39 .
drwxr-xr-x 3 root root 4096 Dec 6 11:39 ..
-rw-r–r– 1 root root 190 Dec 6 11:39 changelog
-rw-r–r– 1 root root 2 Dec 6 11:39 compat
-rw-r–r– 1 root root 525 Dec 6 11:39 control
-rw-r–r– 1 root root 1679 Dec 6 11:39 copyright
-rw-r–r– 1 root root 0 Dec 6 11:39 docs
-rw-r–r– 1 root root 4596 Dec 6 11:39 init.d.ex
-rw-r–r– 1 root root 1646 Dec 6 11:39 manpage.1.ex
-rw-r–r– 1 root root 4663 Dec 6 11:39 manpage.sgml.ex
-rw-r–r– 1 root root 11018 Dec 6 11:39 manpage.xml.ex
-rw-r–r– 1 root root 129 Dec 6 11:39 menu.ex
-rw-r–r– 1 root root 134 Dec 6 11:39 mysource.cron.d.ex
-rw-r–r– 1 root root 238 Dec 6 11:39 mysource.default.ex
-rw-r–r– 1 root root 526 Dec 6 11:39 mysource.doc-base.EX
-rw-r–r– 1 root root 958 Dec 6 11:39 postinst.ex
-rw-r–r– 1 root root 931 Dec 6 11:39 postrm.ex
-rw-r–r– 1 root root 691 Dec 6 11:39 preinst.ex
-rw-r–r– 1 root root 878 Dec 6 11:39 prerm.ex
-rw-r–r– 1 root root 186 Dec 6 11:39 README.Debian
-rw-r–r– 1 root root 271 Dec 6 11:39 README.source
-rwxr-xr-x 1 root root 114 Dec 6 11:39 rules
drwxr-xr-x 2 root root 4096 Dec 6 11:39 source
-rw-r–r– 1 root root 787 Dec 6 11:39 watch.ex

  • You can then modify the relevant files for eg: control

Python: ImportError: /usr/local/lib/python2.7/lib-dynload/_io.so: undefined symbol: PyUnicodeUCS2_Replace

The above error might occur if you have compiled and installed Python from source on Ubuntu.

curl https://www.python.org/ftp/python/2.7.6/Python-2.7.6.tgz -o python.tgz

tar -xvzf python.tgz

cd Python-2.7.6/

./configure

make

make install

 

The error can be fixed by

./configure –enable-unicode=ucs4

Docker – How To Set Locale In Ubuntu

Add the following in your Dockerfile to set Locale in Ubuntu

RUN locale-gen en_US.UTF-8
ENV LANG en_US.UTF-8
ENV LANGUAGE en_US:en
ENV LC_ALL en_US.UTF-8

Jenkins – config.xml

If there is a need to debug Jenkins configuration and the when the interface set up looks correct, it’s better to find more detailed configuration information via the config.xml.

  1. In the server where Jenkins is installed, usually in path /var/lob/jenkins/jobs
  2. Browse to your job folder
  3. There will be a config.xml which will contain the job configuration details

Jenkins – How To Disable Jobs

How To Disable One Jenkins Job

  • Get breadcrumb

curl -u “<user>:<password>“ ‘https://<jenkins_url>/crumbIssuer/api/xml?xpath=concat(//crumbRequestField,”:”,//crumb)’

 

  • curl -I -X POST https://<jenkins_url>/<job_path>/disable –user <user>:<password> -H “<jenkins bread crumb>”

How To Disable Jenkins jobs Under A Folder

  • Via Jenkins script

https://<jenkins_url>/script

  • Enter the following in the script and execute it

folderName=”Build/APP/test” //full name of the folder you want to disable all jobs in

Jenkins.instance.getItemByFullName(folderName).allJobs

.each {

      it.setDisabled(true)

      println(“Disabled job: [$it.fullName]”)

    }

null

Note: Look for file structure in server, ignore the “jobs” path in URL

Docker – How To Source A File In Dockerfile

If you want to source /etc/profile.d/apache-maven.sh, do this in your Dockerfile

RUN echo ‘. /etc/profile.d/apache-maven.sh’ >> ~/.bashrc

Consul – Integrating Nagios Checks

We can integrate script checks into Consul.  To do so with Nagios:

  • Install nagios-plugins-basic. That will contain some basic checks such as disk util, and cpu load

apt-get install nagios-plugins-basic

  • There are also other custom Nagios checks on the internet for eg: checks for Open Connections

Put this script in:

/usr/lib/nagios/plugins

  • Create JSON config (connections.json) and put it under /etc/consul.d/client

{
“check”: {
“name”: “Open Connections”,
“interval”: “60s”,
“args”: [“/usr/lib/nagios/plugins/check_connections”, “-c”, “{{connection_limit_critical}}”, “-w”, “{{connection_limit_warn}}” ],
“status”: “passing”
}

}

  • Restart Consul service
  • You should see this on Consul UI

How To Publish Artifacts To Artifactory Using Curl

We can publish an artifact onto Artifactory using unconventional way such as Curl.

This is how we can do that:

  • Generate a pom file using a bash script, so the artifact will have versioning, etc.
#!/usr/bin/env bash
cat  << EOF
<?xml version="1.0" encoding="UTF-8"?>
<project xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd" xmlns="http://maven.apache.org/POM/4.0.0"
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
  <modelVersion>1.0.0</modelVersion>
  <groupId>com.org.test</groupId>
  <artifactId>apple</artifactId>
  <version>apple</version>
  <packaging>tar</packaging>
</project>
EOF
  • Post onto Artifactory using Curl

curl -i -X PUT -u username:password –data-binary @”apple.tar” “https://artifactory/snapshot/com/org/test/apple/apple.tar”

Starting Dropwizard Application Using Jar Packaging & SysV

By default Debian package will generate an Upstart and SysV script.

This blog will show how we can ship an application in Jar file and start it using SysV

  • Put your jar file somewhere. E.g: /opt
  • Copy the Debian package generated config in /etc into the new app config directory. You might need to reformat the YAML file a bit.

For example:

Some vars might look like this:

timeout: ${sys.TIMEOUT!”5000ms” }

We want to change it to look like:

timeout: 5000ms

  • Uninstall Debian packages

apt-get purge <package>

  • Delete the generated Upstart (/etc/init/<package>.conf) and SysV script (/etc/init.d/<package>)
  • Create a SysV script which looks like this:
#!/bin/bash
#
# dropwizard     This shell script takes care of starting and stopping Dropwizard applications
#
# chkconfig: - 80 20
#
### BEGIN INIT INFO
# Provides: dropwizard
# Required-Start: $network $syslog
# Required-Stop: $network $syslog
# Default-Start:
# Default-Stop:
# Short-Description: start and stop dropwizard
### END INIT INFO

#You just need to replace the {{ var }} below for your application
APPLICATION_NAME="{{ app_name }}"
APPLICATION_USER="{{ app_user }}"
APPLICATION_HOME="{{ app_home }}"
APPLICATION_JAR="{{ app_name }}.jar"
APPLICATION_CONFIG="{{ app_name }}.yml"
APPLICATION_CONFIG_DIR="{{ app_config_dir}}"
APPLICATION_CMD="java {{ additional_java_args }} -jar ${APPLICATION_HOME}/${APPLICATION_JAR} server ${APPLICATION_CONFIG_DIR}/${APPLICATION_CONFIG}"
APPLICATION_SHUTDOWN_WAIT=120

dropwizard_pid() {
    echo `ps aux | grep "${APPLICATION_CMD}" | grep -v grep | awk '{ print $2 }'`
}

start() {
    pid=$(dropwizard_pid)
    if [ -n "$pid" ]
    then
        echo "${APPLICATION_NAME} is already running (pid: $pid)"
    else
        # Start dropwizard
        echo "Starting ${APPLICATION_NAME}"
        su ${APPLICATION_USER} -c "cd ${APPLICATION_HOME}; ${APPLICATION_CMD} > /dev/null &"
    fi
    return 0
}

stop() {
    pid=$(dropwizard_pid)
    if [ -n "$pid" ]
    then
        echo "Stopping ${APPLICATION_NAME}"
        kill $pid

    let kwait=$APPLICATION_SHUTDOWN_WAIT
    count=0
    count_by=5
    until [ `ps -p $pid | grep -c $pid` = '0' ] || [ $count -gt $kwait ]
    do
        echo "Waiting for processes to exit. Timeout before we kill the pid: ${count}/${kwait}"
        sleep $count_by
        let count=$count+$count_by;
    done

    if [ $count -gt $kwait ]; then
        echo "Killing processes which didn't stop after ${APPLICATION_SHUTDOWN_WAIT} seconds"
        kill -9 $pid
    fi
    else
        echo "${APPLICATION_NAME} is not running"
    fi

    return 0
}

status(){
    pid=$(dropwizard_pid)
    if [ -n "$pid" ]; then
        echo "${APPLICATION_NAME} is running with pid: $pid"
    else
        echo "${APPLICATION_NAME} is not running"
    exit 1
    fi
}


case $1 in
    start)
        start
        ;;
    stop)
        stop
        ;;
    restart)
        stop
        start
        ;;
    status)
        status
        ;;
    *)
    echo "Usage: $0 {start|stop|restart|status}"
    exit 1
    ;;

esac

exit 0
  • Finally, we can start the application by:

service <package> start

Jenkinsfile – Credentials Binding Plugin – sshUserPrivateKey

Example below shows how to use the sshUserPrivateKey of a server to do Git clone in Jenkinsfile:

node(‘jenkins_node’){

stage(‘checkout code’){

//Private key for server is stored in Jenkins with id ‘private_key_for_server’ and can be accessible via variable ‘private_key’

withCredentials([sshUserPrivateKey(credentialsId: ‘private_key_for _server’, keyFileVariable: ‘private_key’, passphraseVariable: ”, usernameVariable: ”)]){

// start ssh-agent
sh ‘ssh-agent /bin/bash’

// add private key to ssh-agent, check if private key is successfully added and git clone using the private key

sh ‘eval $(ssh-agent) && ssh-add ${private_key} && ssh-add -l && git clone git@git.test.com:test.git’

}

}

}

 

« Older posts

© 2019 Chuan Chuan Law

Theme by Anders NorenUp ↑