CouchBase Issues / Errors - Over coming them..

ISSUE / ERROR # 1
Attention - Join completion call failed. Got HTTP status 500 from REST call post to http://hostname.domain.com:8091/completeJoin. Body was: "[\"Unexpected server error, request logged.\"]"

AIM : Adding Nodes to a Cluster
ERROR: Attention - Join completion call failed. Got HTTP status 500 from REST call post to http://hostname.domain.com:8091/completeJoin. Body was: "[\"Unexpected server error, request logged.\"]"
REASON (RCA): When you add a Node with already existing bucket to a cluster, You will face this issue. There can be other factors, but in my case, i found this as the issue.
How to over come this ? Follow the below steps: But please consult your team before that as some commands below are ab
solutely destructive if you are not paying attention.

1. stop the couchbase first
service couchbase status
service couchbase stop

2. Check that the ports are not being used by anybody, if they are, trace the program using it and kill it.
netstat -a |grep 8091
netstat -a |grep 8092
netstat -a |grep 11211
netstat -a |grep 11210

3. Move/Remove the data directories - your choice
cd /data/couchbase
mv data data.OLD
mv index index.OLD

4. Recreate them
mkdir data index

5. Assign them to the couchbase user

chown couchbase:couchbase data index

6. Once again check on the ports
netstat -a |grep 8091
netstat -a |grep 8092
netstat -a |grep 11211
netstat -a |grep 11210
Usually 11210 will be engaged by some program, hence
wait until 11210 disappears

7. If all good, go ahead and start the couchbase
service couchbase status
service couchbase start

8. Now you can go to the master node and add it to the Server Groups..


ISSUE / ERROR # 2 - Couchbase Server alert: auto_failover_node

Scenario 1a:
Port server memcached on node 'babysitter_of_ns_1@127.0.0.1' exited with status 0.
If the exit status is zero, mostly due to one of your DBAs is working not that instance and might have brought down the couch base gracefully.
Just Start the couch base and add it back in the server list and you should be good to go.

Scenario 1b:
Port server memcached on node 'babysitter_of_ns_1@127.0.0.1' exited with status 0.
There can be many reasons why a couch base server will be failed over and the exit status can be .
- Because of the network glitches and if you have the Auto-Failover time out set to a too low number such as 30, couch base server manager may think that the box is down when it is not reachable.

Scenario 2:
Port server moxi on node 'babysitter_of_ns_1@127.0.0.1' exited with status134
Explanation will be provided soon..

Exploring Couchbase Via Command Line

Exploring Couchbase

Few links to get familiarize with :
1. You can get the complete information on the Couchbase tools and CLI @
http://docs.couchbase.com/couchbase-manual-2.5/cb-cli/
2. Fab
http://docs.fabfile.org/en/latest/usage/fab.html


DAEMON=/opt/couchbase/bin/couchbase-server
PIDFILE=/opt/couchbase/var/lib/couchbase/couchbase-server.pid
NODEFILE=/opt/couchbase/var/lib/couchbase/couchbase-server.node
COOKIEFILE=/opt/couchbase/var/lib/couchbase/couchbase-server.cookie





How to Un-Install Couchbase ?
If you have used rpms, then follow the below (of course every firm has its own way)

1. Check the status of the couchbase and stop if it is running..
 service couchbase status
 service couchbase stop
2. Check the Rpms installed
 rmp -qa | grep couchbase
3. Remove the Rpms
 rmp -e couchbase-server-2.5-1083.x86_64
4. Clean up the related directories (if you have different ones, then proceed accordingly)
NOTE: Becareful, below are destructive (if you have other files, then just remove couchbase related files)
 cd /opt
 rm -rf *
 cd /data/couchbase
 rm -rf *
 ls -ltr /etc/init.d/couch* -- shouldn't see anything

Change the couch base user password ?

Note: Point to remember is : Nothing goes wrong when you change the password, but at the same time you should change the password on all the nodes in a cluster.

 pwd
/opt/couchbase/bin
[root@cbhostraj bin]# ./cbreset_password

Please enter the new administrative password (or <Enter> for system generated password):

Running this command will reset administrative password.
Do you really want to do it? (yes/no)yes
Resetting administrative password...
Password for user admin was successfully replaced.

My New Art Projects

                                                           My Abstract Paintings.. 

Painting was my favorite.. But stopped it as soon as I entered my 10th Standard and until now, I have never touched the colors, paint brush or canvas. After a long time, I recently started working on it but soon realized that the skill for holding paint brush became pretty difficult for me. Meanwhile i am working on that, I started on the Abstract painting with Acrylics and I am loving it..

Bright emotion of Lava 

(Inspired by Brigitte König)

Ganesh on my Abstract Paint..

CouchBase Installation on Ubuntu

Couchbase Installation..

Couchbase installation is pretty straight forward.

go to http://www.couchbase.com/download and based on your operating system & Config, download the required version - Typically around 150 MB of size.
you can download it to the machine using sftp as shown below.

Note : for Red Hat Linux Systems, thing to consider is : THP (Transparent huge pages) should be disabled which is recommend by the couch base.
/* Check Status

You can check the current status of THP with the following command:

cat /sys/kernel/mm/*transparent_hugepage/enabled

Output will resemble the following, with the current status marked in [brackets]:

[always] madvise never

 Always indicates that THP will be used for all allocations, and Couchbase recommends disabling THP in this instance. Note: Older kernels may only show the always and never options. 

You can check the current status of THP defrag with the following command:

cat /sys/kernel/mm/*transparent_hugepage/defrag

Output will resemble the following, where always is specified, meaning that THP defrag is always enabled:

[always] never

Note: some distributions name the flag transparent_hugepage, others redhat_transparent_hugepage, hence the wildcard). */ 

For more info, please visit : http://support.couchbase.com/entries/22160929-Transparent-Huge-Pages

UbantoAz:Downloads raj$ sftp 10.64.211.111

rmattewada@10.64.211.238's password: 

Connected to 10.64.211.111.

sftp> cd /tmp

sftp> pwd

Remote working directory: /tmp

sftp> put couchbase-server-enterprise_2.5.1_x86_64.deb  .

Uploading couchbase-server-enterprise_2.5.1_x86_64.deb to /tmp/./couchbase-server-enterprise_2.5.1_x86_64.deb

couchbase-server-enterprise_2.5.1_x86_64.deb                                                                                    100%  134MB   5.6MB/s   00:24    

sftp> ls

couchbase-server-enterprise_2.5.1_x86_64.deb          

Once the copy is done, you can follow below as root user:
Note that in Ubuntu, you have Debian as package manager, where as in RHEL, you will see yum.

root@csshtest-6001:/opt# pwd

/opt

root@csshtest-6001:/opt# cp /tmp/couchbase-server-enterprise_2.5.1_x86_64.deb .

root@csshtest-6001:/opt# uname -a

Linux csshtest-6001 3.13.0-24-generic #47-Ubuntu SMP Fri May 2 23:30:00 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux

root@csshtest-6001:/opt# #For Ubuntu version 12.04, you need to install a specific OpenSSL dependency by running:

root@csshtest-6001:/opt# id

uid=0(root) gid=0(root) groups=0(root)

root@csshtest-6001:/opt# apt-get install libssl0.9.8

Reading package lists... Done

Building dependency tree       

Reading state information... Done

The following NEW packages will be installed:

  libssl0.9.8

0 upgraded, 1 newly installed, 0 to remove and 87 not upgraded.

Need to get 692 kB of archives.

After this operation, 2,445 kB of additional disk space will be used.

Get:1 http://installsvc.vip/mirrors/archive.ubuntu.com/ubuntu/ trusty-updates/universe libssl0.9.8 amd64 0.9.8o-7ubuntu3.2.14.04.1 [692 kB]

Fetched 692 kB in 0s (4,135 kB/s)

Preconfiguring packages ...

Selecting previously unselected package libssl0.9.8:amd64.

(Reading database ... 56731 files and directories currently installed.)

Preparing to unpack .../libssl0.9.8_0.9.8o-7ubuntu3.2.14.04.1_amd64.deb ...

Unpacking libssl0.9.8:amd64 (0.9.8o-7ubuntu3.2.14.04.1) ...

Setting up libssl0.9.8:amd64 (0.9.8o-7ubuntu3.2.14.04.1) ...

Processing triggers for libc-bin (2.19-0ubuntu6) ...

root@csshtest-6001:/opt# 

root@csshtest-6001:/opt# #The Ubuntu Couchbase installation uses the DEB package. To install, use the dpkg command-line tool using the DEB file that you downloaded. The following example uses sudo which will require root-access to allow installation:

root@csshtest-6001:/opt# 

root@csshtest-6001:/opt# mkdir couchbase

root@csshtest-6001:/opt# cd couchbase/

root@csshtest-6001:/opt/couchbase# cp ../couchbase

couchbase/                                    couchbase-server-enterprise_2.5.1_x86_64.deb  

root@csshtest-6001:/opt/couchbase# cp ../couchbase-server-enterprise_2.5.1_x86_64.deb  .

root@csshtest-6001:/opt/couchbase# 

root@csshtest-6001:/opt/couchbase# dpkg -i couchbase-server-enterprise_2.5.1_x86_64.deb 

Selecting previously unselected package couchbase-server.

(Reading database ... 56755 files and directories currently installed.)

Preparing to unpack couchbase-server-enterprise_2.5.1_x86_64.deb ...

libssl1* is installed. Continue installing

Minimum RAM required  : 4 GB

System RAM configured : 1017948 kB

Minimum number of processors required : 4 cores

Number of processors on the system    : 1 cores

Unpacking couchbase-server (2.5.1) ...

Setting up couchbase-server (2.5.1) ...

 * Started couchbase-server

You have successfully installed Couchbase Server.

Please browse to http://csshtest-6001:8091/ to configure your server.

Please refer to http://couchbase.com for additional resources.

Please note that you have to update your firewall configuration to

allow connections to the following ports: 11211, 11210, 11209, 4369,

8091, 8092, 18091, 18092, 11214, 11215 and from 21100 to 21299.

By using this software you agree to the End User License Agreement.

See /opt/couchbase/LICENSE.txt.

Processing triggers for ureadahead (0.100.0-16) ...

root@csshtest-6001:/opt/couchbase# 

Now, open a browser and type in : http://ip_address:8091
you will get the set up prompt, click on that and you will see

decide on per server Ram Quota - Typically 75% of the total available RAM or
80% of the total available RAM if you are using XDCR (Cross Data Center Replication)
Note: you can join a cluster immediately (if you are planning to do so) if you check that instead of starting as a new cluster.

If you want to play with the sample data, check in for available samples like Beer or gamesim.

Depending on your I/O Load, chose the reader/sriter workers
Do not enable Flush if you are using this for production.
decide on per node ram Quota for bucket and allocate one.

pass through the next step and at step 5, provide username, password and you are done.

You can follow the same for all the servers in your cluster and bring them up. Then, you can add the nodes to a cluster.

Observations:

Observations:
1. when there are 3 nodes in a cluster,
if 2 nodes are down and another node is added, but in pending balance state,
and in that state, if that server server goes down, the admin console says:
Servers Down: 2
Servers Pending Rebalance: 1
where as i think it should say Servers Down: 3

2. when rebalance is in progress, if we click on the stop rebalance, the operation would not stop
though it says "rebalance stopped by janitar"
to verify that, in the server nodes tab, go to pending rebalance and you will not notice anything pending.