发表于 | 分类于 | 0 | 阅读次数 2471

Cloudera Manager安装

centos7,MySQL

安装前的准备：

  1. 为集群中每一个主机配置hostname
sudo hostnamectl set-hostname foo-1.example.com
vi /etc/hosts

添加

1.1.1.1  foo-1.example.com  foo-1
2.2.2.2  foo-2.example.com  foo-2
3.3.3.3  foo-3.example.com  foo-3
4.4.4.4  foo-4.example.com  foo-4

vi /etc/sysconfig/network
HOSTNAME=foo-1.example.com

  1. 关闭防火墙
    systemctl stop firewalld

  2. 关闭SELinux
    临时修改:setenforce 0
    永久修改:
    修改/etc/selinux/config中SELINUX=enforcing为SELINUX=permissive
    通过执行getenforce获取SELinux状态

  3. 时钟同步
    yum install ntp

编辑/etc/ntp.conf,例如增加以下:
server 0.pool.ntp.org
server 1.pool.ntp.org
server 2.pool.ntp.org

启动ntp服务
systemctl start ntp
加入开机自启
systemctl enable ntpd
同步节点
ntpdate -u <ntp_server>
同步系统时钟
hwclock --systohc

  1. 检测Python2.7是否安装成功
    python --version
    如果没有则需要安装Python2.7
    yum install python27

安装过程

配置Cloudera Manager存储库

  1. 将对应版本的cloudera-manager.repo下载到/etc/yum.repos.d/
    例如：sudo wget <repo_file_url> -P /etc/yum.repos.d/，<repo_file_url> 为repo文件下载路径，如https://archive.cloudera.com/cm6/6.1.0/redhat6/yum/cloudera-manager.repo
  2. 导入存储库签名密钥
    sudo rpm --import https://archive.cloudera.com/cm6/6.1.0/redhat7/yum/RPM-GPG-KEY-cloudera

安装JDK

  1. JDK必须是64位,并且最好是Oracle JDK1.8

  2. JDK安装路径需在 /usr/java/jdk-version

  3. 除手动安装JDK外，也可使用 CLoudera Manager存储库安装
    sudo yum install oracle-j2sdk1.8

  4. 可以使用如下命令自动完成(将jdk1.8.0_141-cloudera替换为自己的jdk安装文件夹)
    sudo JAVA_HOME=/usr/java/jdk1.8.0_141-cloudera /opt/cloudera/cm-agent/bin/certmanager setup --configure-services

安装Cloudera Manager Server

  • sudo yum install cloudera-manager-daemons cloudera-manager-agent cloudera-manager-server（只有主节点需要安装三个，其余节点不需要cloudera-manager-server）
  • 或者下载rpm包到本地 使用rpm命令手动安装（daemons比较大，大概1G多，可以直接安装完成，agent有其他依赖而且比较小，建议使用yum安装）。
下载CM6.1的安装包
https://archive.cloudera.com/cm6/6.1.0/redhat7/yum/RPMS/x86_64/cloudera-manager-agent-6.1.0-769885.el7.x86_64.rpm
https://archive.cloudera.com/cm6/6.1.0/redhat7/yum/RPMS/x86_64/cloudera-manager-daemons-6.1.0-769885.el7.x86_64.rpm
https://archive.cloudera.com/cm6/6.1.0/redhat7/yum/RPMS/x86_64/cloudera-manager-server-6.1.0-769885.el7.x86_64.rpm
https://archive.cloudera.com/cm6/6.1.0/redhat7/yum/RPMS/x86_64/cloudera-manager-server-db-2-6.1.0-769885.el7.x86_64.rpm
https://archive.cloudera.com/cm6/6.1.0/redhat7/yum/RPMS/x86_64/oracle-j2sdk1.8-1.8.0+update141-1.x86_64.rpm
https://archive.cloudera.com/cm6/6.1.0/allkeys.asc
 
下载CDH6.1的安装包
https://archive.cloudera.com/cdh6/6.1.0/parcels/CDH-6.1.0-1.cdh6.1.0.p0.770702-el7.parcel
https://archive.cloudera.com/cdh6/6.1.0/parcels/CDH-6.1.0-1.cdh6.1.0.p0.770702-el7.parcel.sha256
https://archive.cloudera.com/cdh6/6.1.0/parcels/manifest.json
 
CDH6.1.0 rpm软件包下载
https://archive.cloudera.com/cdh6/6.1.0/redhat7/yum/RPMS/noarch/
https://archive.cloudera.com/cdh6/6.1.0/redhat7/yum/RPMS/x86_64/

  • 启用Auto-TLS
    可以后续在Web UI界面完成。

  • 检测是否正确安装
    检验日志 /var/log/cloudera-scm-agent/certmanager.log 确保 /var/lib/cloudera-scm-server/certmanager/*已被创建

安装数据库

支持MariaDB,MySQL,PostgreSQL,Oracle
这里使用MySQL.

  1. 下载
    wget http://repo.mysql.com/mysql-community-release-el7-5.noarch.rpm
  2. 安装
    sudo rpm -ivh mysql-community-release-el7-5.noarch.rpm
    sudo yum update
    sudo yum install mysql-server
  3. 测试能否启动MySQL服务
    sudo systemctl start mysqld
  4. 将 /var/lib/mysql/ib_logfile0 和 /var/lib/mysql/ib_logfile1移动到/var/lib/mysql做一个备份
  5. 编辑my.cnf(默认在/etc/my.cnf)
    Cloudera Manager建议的my.cnf配置如下:
[mysqld]
datadir=/var/lib/mysql
socket=/var/lib/mysql/mysql.sock
transaction-isolation = READ-COMMITTED
# Disabling symbolic-links is recommended to prevent assorted security risks;
# to do so, uncomment this line:
symbolic-links = 0

key_buffer_size = 32M
max_allowed_packet = 32M
thread_stack = 256K
thread_cache_size = 64
query_cache_limit = 8M
query_cache_size = 64M
query_cache_type = 1

max_connections = 550
#expire_logs_days = 10
#max_binlog_size = 100M

#log_bin should be on a disk with enough free space.
#Replace '/var/lib/mysql/mysql_binary_log' with an appropriate path for your
#system and chown the specified folder to the mysql user.
log_bin=/var/lib/mysql/mysql_binary_log

#In later versions of MySQL, if you enable the binary log and do not set
#a server_id, MySQL will not start. The server_id must be unique within
#the replicating group.
server_id=1

binlog_format = mixed

read_buffer_size = 2M
read_rnd_buffer_size = 16M
sort_buffer_size = 8M
join_buffer_size = 8M

# InnoDB settings
innodb_file_per_table = 1
innodb_flush_log_at_trx_commit  = 2
innodb_log_buffer_size = 64M
innodb_buffer_pool_size = 4G
innodb_thread_concurrency = 8
innodb_flush_method = O_DIRECT
innodb_log_file_size = 512M

[mysqld_safe]
log-error=/var/log/mysqld.log
pid-file=/var/run/mysqld/mysqld.pid

sql_mode=STRICT_ALL_TABLES

  1. 为MySQL设置开机自启
    sudo systemctl enable mysqld

  2. 设置MySQL密码
    sudo /usr/bin/mysql_secure_installation
    加粗部分为需要输入的内容
    [...]
    Enter current password for root (enter for none):
    OK, successfully used password, moving on...
    [...]
    Set root password? [Y/n] Y
    New password:
    Re-enter new password:
    Remove anonymous users? [Y/n] Y
    [...]
    Disallow root login remotely? [Y/n] N
    [...]
    Remove test database and access to it [Y/n] Y
    [...]
    Reload privilege tables now? [Y/n] Y
    All done!

  3. 下载jdbc驱动
    wget https://dev.mysql.com/get/Downloads/Connector-J/mysql-connector-java-5.1.46.tar.gz

  4. 在MySQL中创建Cloudera Manager中需要用到的数据库及用户
    MySQL中的编码格式一定要是utf8

mysql -u root -p登录MySQL。

使用CREATE DATABASE <database> DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;创建数据库

使用GRANT ALL ON <database>.* TO '<user>'@'%' IDENTIFIED BY '<password>';新建用户

其中password需自己修改,这里是123456

CREATE DATABASE scm DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
CREATE DATABASE amon DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
CREATE DATABASE rman DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
CREATE DATABASE hue DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
CREATE DATABASE metastore DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
CREATE DATABASE sentry DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
CREATE DATABASE nav DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
CREATE DATABASE navms DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
CREATE DATABASE oozie DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
GRANT ALL ON scm.* TO 'scm'@'%' IDENTIFIED BY '123456';
GRANT ALL ON amon.* TO 'amon'@'%' IDENTIFIED BY '123456';
GRANT ALL ON rman.* TO 'rman'@'%' IDENTIFIED BY '123456';
GRANT ALL ON hue.* TO 'hue'@'%' IDENTIFIED BY '123456';
GRANT ALL ON metastroe.* TO 'hive'@'%' IDENTIFIED BY '123456';
GRANT ALL ON sentry.* TO 'sentry'@'%' IDENTIFIED BY '123456';
GRANT ALL ON nav.* TO 'nav'@'%' IDENTIFIED BY '123456';
GRANT ALL ON navms.* TO 'navms'@'%' IDENTIFIED BY '123456';
GRANT ALL ON oozie.* TO 'oozie'@'%' IDENTIFIED BY '123456';

各数据库对应的角色

服务数据库用户名
Cloudera Manager Serverscmscm
Activity Monitoramonamon
Reports Managerrmanrman
Huehuehue
Hive Metastore Servermetastorehive
Sentry Serversentrysentry
Cloudera Navigator Audit Servernavnav
Cloudera Navigator Metadata Servernavmsnavms
Oozieoozieoozie

Cloudera Manager数据库初始化

  1. 使用命令sudo /opt/cloudera/cm/schema/scm_prepare_database.sh [options] <databaseType> <databaseName> <databaseUser> <password>
    其中为mysql,其余三项是之前配置的。例如sudo /opt/cloudera/cm/schema/scm_prepare_database.sh mysql scm 123456，仅执行一次即可，此步骤是为配置Cloudera Manager使用的数据库

安装CDH及其他软件

启动CLoudera Manager Server
sudo systemctl start cloudera-scm-server
使用以下命令查看启动日志
sudo tail -f /var/log/cloudera-scm-server/cloudera-scm-server.log
当看到如下内容说明启动成功

INFO WebServerImpl:com.cloudera.server.cmf.WebServerImpl: Started Jetty server.

通过ip:7180端口即可访问(如果配置了Auto-TLS则是https://ip:7183)
用户名与密码均为admin

登录成功后添加群集,主机,服务.

添加服务

在线安装

点击主机面右上角的礼物界面，进入到parcel管理界面，在左边选中需要添加服务的集群，右边点击需要下载的服务，等待下载完成之后分发，激活。

离线安装

将下载的parcel文件，sha文件，manifest.json，文件放在运行有cloudera-manager-server主机的/opt/cloudera/parcel-repo/ 文件夹下，

添加角色实例

在集群管理界面点击添加服务 -> 自定义服务 ->选择需要添加的服务以及主机节点。

添加主机

  1. 配置主机的hosts,安装ntp，
  2. 手动或在线安装cloudera-manager-daemons服务。
  3. 点击需要添加主机的集群后，点击Add Hosts，输入IP地址，安装parcel包即可。

需要注意的坑

  1. SSL证书的域名需要与运行cloudera-manager-server的域名相同，因为用cloudera-manager的TLS脚本可以自动生成，并且生成后较难修改，所以需要注意。
  2. 因网站是国外的，所以下载rpm包比较费时，可以将下载过的rpm包保存在本地，并启用httpd，制作本地yum源下载。
  3. 如若遇到将parcel文件以及sha文件放在/opt/cloudera/parcel-repo/文件夹下，哈希校验失败的情况，需要使用sha1sum或sha256sum对parcel文件进行验证，确定无误的话，可以手动将sha256改为sha，并把结果写进.parcel.sha文件中。

参考文档

CSDN：离线安装CDH6.1.0

cnblogs:CDH6安装过程

Clodera Manager官方安装文档