写BUG的派大星

Patrick Star

  • 首页
  • 归档

  • 搜索
设计模式 Gis Kafka Druid 微信小程序 Java 开源项目源码 物体识别 机器学习 Mybatis 微服务 Feign OpenVPN CSS Streamsets CDH SpringCloud SpringBoot maven 分布式 Shell Tree Linux js WebSocket 多线程 集群 Hadoop 大数据 JDK ElasticSearch MySQL 数据库 Redis Http Nginx

Cloudera Manager安装与搭建(CentOS7,CDH6.1.0)

发表于 2020-06-08 | 分类于 大数据 | 0 | 阅读次数 2272

Cloudera Manager安装

centos7,MySQL

安装前的准备:

  1. 为集群中每一个主机配置hostname
sudo hostnamectl set-hostname foo-1.example.com
vi /etc/hosts

添加

1.1.1.1  foo-1.example.com  foo-1
2.2.2.2  foo-2.example.com  foo-2
3.3.3.3  foo-3.example.com  foo-3
4.4.4.4  foo-4.example.com  foo-4

vi /etc/sysconfig/network
HOSTNAME=foo-1.example.com

  1. 关闭防火墙
    systemctl stop firewalld

  2. 关闭SELinux
    临时修改:setenforce 0
    永久修改:
    修改/etc/selinux/config中SELINUX=enforcing为SELINUX=permissive
    通过执行getenforce获取SELinux状态

  3. 时钟同步
    yum install ntp

编辑/etc/ntp.conf,例如增加以下:
server 0.pool.ntp.org
server 1.pool.ntp.org
server 2.pool.ntp.org

启动ntp服务
systemctl start ntp
加入开机自启
systemctl enable ntpd
同步节点
ntpdate -u <ntp_server>
同步系统时钟
hwclock --systohc

  1. 检测Python2.7是否安装成功
    python --version
    如果没有则需要安装Python2.7
    yum install python27

安装过程

配置Cloudera Manager存储库

  1. 将对应版本的cloudera-manager.repo下载到/etc/yum.repos.d/
    例如:sudo wget <repo_file_url> -P /etc/yum.repos.d/,<repo_file_url> 为repo文件下载路径,如https://archive.cloudera.com/cm6/6.1.0/redhat6/yum/cloudera-manager.repo
  2. 导入存储库签名密钥
    sudo rpm --import https://archive.cloudera.com/cm6/6.1.0/redhat7/yum/RPM-GPG-KEY-cloudera

安装JDK

  1. JDK必须是64位,并且最好是Oracle JDK1.8

  2. JDK安装路径需在 /usr/java/jdk-version

  3. 除手动安装JDK外,也可使用 CLoudera Manager存储库安装
    sudo yum install oracle-j2sdk1.8

  4. 可以使用如下命令自动完成(将jdk1.8.0_141-cloudera替换为自己的jdk安装文件夹)
    sudo JAVA_HOME=/usr/java/jdk1.8.0_141-cloudera /opt/cloudera/cm-agent/bin/certmanager setup --configure-services

安装Cloudera Manager Server

  • sudo yum install cloudera-manager-daemons cloudera-manager-agent cloudera-manager-server(只有主节点需要安装三个,其余节点不需要cloudera-manager-server)
  • 或者下载rpm包到本地 使用rpm命令手动安装(daemons比较大,大概1G多,可以直接安装完成,agent有其他依赖而且比较小,建议使用yum安装)。
下载CM6.1的安装包
https://archive.cloudera.com/cm6/6.1.0/redhat7/yum/RPMS/x86_64/cloudera-manager-agent-6.1.0-769885.el7.x86_64.rpm
https://archive.cloudera.com/cm6/6.1.0/redhat7/yum/RPMS/x86_64/cloudera-manager-daemons-6.1.0-769885.el7.x86_64.rpm
https://archive.cloudera.com/cm6/6.1.0/redhat7/yum/RPMS/x86_64/cloudera-manager-server-6.1.0-769885.el7.x86_64.rpm
https://archive.cloudera.com/cm6/6.1.0/redhat7/yum/RPMS/x86_64/cloudera-manager-server-db-2-6.1.0-769885.el7.x86_64.rpm
https://archive.cloudera.com/cm6/6.1.0/redhat7/yum/RPMS/x86_64/oracle-j2sdk1.8-1.8.0+update141-1.x86_64.rpm
https://archive.cloudera.com/cm6/6.1.0/allkeys.asc
 
下载CDH6.1的安装包
https://archive.cloudera.com/cdh6/6.1.0/parcels/CDH-6.1.0-1.cdh6.1.0.p0.770702-el7.parcel
https://archive.cloudera.com/cdh6/6.1.0/parcels/CDH-6.1.0-1.cdh6.1.0.p0.770702-el7.parcel.sha256
https://archive.cloudera.com/cdh6/6.1.0/parcels/manifest.json
 
CDH6.1.0 rpm软件包下载
https://archive.cloudera.com/cdh6/6.1.0/redhat7/yum/RPMS/noarch/
https://archive.cloudera.com/cdh6/6.1.0/redhat7/yum/RPMS/x86_64/
  • 启用Auto-TLS
    可以后续在Web UI界面完成。

  • 检测是否正确安装
    检验日志 /var/log/cloudera-scm-agent/certmanager.log 确保 /var/lib/cloudera-scm-server/certmanager/*已被创建

安装数据库

支持MariaDB,MySQL,PostgreSQL,Oracle
这里使用MySQL.

  1. 下载
    wget http://repo.mysql.com/mysql-community-release-el7-5.noarch.rpm
  2. 安装
    sudo rpm -ivh mysql-community-release-el7-5.noarch.rpm
    sudo yum update
    sudo yum install mysql-server
  3. 测试能否启动MySQL服务
    sudo systemctl start mysqld
  4. 将 /var/lib/mysql/ib_logfile0 和 /var/lib/mysql/ib_logfile1移动到/var/lib/mysql做一个备份
  5. 编辑my.cnf(默认在/etc/my.cnf)
    Cloudera Manager建议的my.cnf配置如下:
[mysqld]
datadir=/var/lib/mysql
socket=/var/lib/mysql/mysql.sock
transaction-isolation = READ-COMMITTED
# Disabling symbolic-links is recommended to prevent assorted security risks;
# to do so, uncomment this line:
symbolic-links = 0

key_buffer_size = 32M
max_allowed_packet = 32M
thread_stack = 256K
thread_cache_size = 64
query_cache_limit = 8M
query_cache_size = 64M
query_cache_type = 1

max_connections = 550
#expire_logs_days = 10
#max_binlog_size = 100M

#log_bin should be on a disk with enough free space.
#Replace '/var/lib/mysql/mysql_binary_log' with an appropriate path for your
#system and chown the specified folder to the mysql user.
log_bin=/var/lib/mysql/mysql_binary_log

#In later versions of MySQL, if you enable the binary log and do not set
#a server_id, MySQL will not start. The server_id must be unique within
#the replicating group.
server_id=1

binlog_format = mixed

read_buffer_size = 2M
read_rnd_buffer_size = 16M
sort_buffer_size = 8M
join_buffer_size = 8M

# InnoDB settings
innodb_file_per_table = 1
innodb_flush_log_at_trx_commit  = 2
innodb_log_buffer_size = 64M
innodb_buffer_pool_size = 4G
innodb_thread_concurrency = 8
innodb_flush_method = O_DIRECT
innodb_log_file_size = 512M

[mysqld_safe]
log-error=/var/log/mysqld.log
pid-file=/var/run/mysqld/mysqld.pid

sql_mode=STRICT_ALL_TABLES
  1. 为MySQL设置开机自启
    sudo systemctl enable mysqld

  2. 设置MySQL密码
    sudo /usr/bin/mysql_secure_installation
    加粗部分为需要输入的内容
    [...]
    Enter current password for root (enter for none):
    OK, successfully used password, moving on...
    [...]
    Set root password? [Y/n] Y
    New password:
    Re-enter new password:
    Remove anonymous users? [Y/n] Y
    [...]
    Disallow root login remotely? [Y/n] N
    [...]
    Remove test database and access to it [Y/n] Y
    [...]
    Reload privilege tables now? [Y/n] Y
    All done!

  3. 下载jdbc驱动
    wget https://dev.mysql.com/get/Downloads/Connector-J/mysql-connector-java-5.1.46.tar.gz

  4. 在MySQL中创建Cloudera Manager中需要用到的数据库及用户
    MySQL中的编码格式一定要是utf8

mysql -u root -p登录MySQL。

使用CREATE DATABASE <database> DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;创建数据库

使用GRANT ALL ON <database>.* TO '<user>'@'%' IDENTIFIED BY '<password>';新建用户

其中password需自己修改,这里是123456

CREATE DATABASE scm DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
CREATE DATABASE amon DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
CREATE DATABASE rman DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
CREATE DATABASE hue DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
CREATE DATABASE metastore DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
CREATE DATABASE sentry DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
CREATE DATABASE nav DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
CREATE DATABASE navms DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
CREATE DATABASE oozie DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
GRANT ALL ON scm.* TO 'scm'@'%' IDENTIFIED BY '123456';
GRANT ALL ON amon.* TO 'amon'@'%' IDENTIFIED BY '123456';
GRANT ALL ON rman.* TO 'rman'@'%' IDENTIFIED BY '123456';
GRANT ALL ON hue.* TO 'hue'@'%' IDENTIFIED BY '123456';
GRANT ALL ON metastroe.* TO 'hive'@'%' IDENTIFIED BY '123456';
GRANT ALL ON sentry.* TO 'sentry'@'%' IDENTIFIED BY '123456';
GRANT ALL ON nav.* TO 'nav'@'%' IDENTIFIED BY '123456';
GRANT ALL ON navms.* TO 'navms'@'%' IDENTIFIED BY '123456';
GRANT ALL ON oozie.* TO 'oozie'@'%' IDENTIFIED BY '123456';

各数据库对应的角色

服务数据库用户名
Cloudera Manager Serverscmscm
Activity Monitoramonamon
Reports Managerrmanrman
Huehuehue
Hive Metastore Servermetastorehive
Sentry Serversentrysentry
Cloudera Navigator Audit Servernavnav
Cloudera Navigator Metadata Servernavmsnavms
Oozieoozieoozie

Cloudera Manager数据库初始化

  1. 使用命令sudo /opt/cloudera/cm/schema/scm_prepare_database.sh [options] <databaseType> <databaseName> <databaseUser> <password>
    其中为mysql,其余三项是之前配置的。例如sudo /opt/cloudera/cm/schema/scm_prepare_database.sh mysql scm 123456,仅执行一次即可,此步骤是为配置Cloudera Manager使用的数据库

安装CDH及其他软件

启动CLoudera Manager Server
sudo systemctl start cloudera-scm-server
使用以下命令查看启动日志
sudo tail -f /var/log/cloudera-scm-server/cloudera-scm-server.log
当看到如下内容说明启动成功

INFO WebServerImpl:com.cloudera.server.cmf.WebServerImpl: Started Jetty server.

通过ip:7180端口即可访问(如果配置了Auto-TLS则是https://ip:7183)
用户名与密码均为admin

登录成功后添加群集,主机,服务.

添加服务

在线安装

点击主机面右上角的礼物界面,进入到parcel管理界面,在左边选中需要添加服务的集群,右边点击需要下载的服务,等待下载完成之后分发,激活。

离线安装

将下载的parcel文件,sha文件,manifest.json,文件放在运行有cloudera-manager-server主机的/opt/cloudera/parcel-repo/ 文件夹下,

添加角色实例

在集群管理界面点击添加服务 -> 自定义服务 ->选择需要添加的服务以及主机节点。

添加主机

  1. 配置主机的hosts,安装ntp,
  2. 手动或在线安装cloudera-manager-daemons服务。
  3. 点击需要添加主机的集群后,点击Add Hosts,输入IP地址,安装parcel包即可。

需要注意的坑

  1. SSL证书的域名需要与运行cloudera-manager-server的域名相同,因为用cloudera-manager的TLS脚本可以自动生成,并且生成后较难修改,所以需要注意。
  2. 因网站是国外的,所以下载rpm包比较费时,可以将下载过的rpm包保存在本地,并启用httpd,制作本地yum源下载。
  3. 如若遇到将parcel文件以及sha文件放在/opt/cloudera/parcel-repo/文件夹下,哈希校验失败的情况,需要使用sha1sum或sha256sum对parcel文件进行验证,确定无误的话,可以手动将sha256改为sha,并把结果写进.parcel.sha文件中。

参考文档

CSDN:离线安装CDH6.1.0

cnblogs:CDH6安装过程

Clodera Manager官方安装文档

  • 本文作者: Patrick
  • 本文链接: https://www.write1bug.cn/archives/clouderamanager安装centos7cdh610
  • 版权声明: 本博客所有文章除特别声明外,均采用CC BY-NC-SA 3.0 许可协议。转载请注明出处!
# 设计模式 # Gis # Kafka # Druid # 微信小程序 # Java # 开源项目源码 # 物体识别 # 机器学习 # Mybatis # 微服务 # Feign # OpenVPN # CSS # Streamsets # CDH # SpringCloud # SpringBoot # maven # 分布式 # Shell # Tree # Linux # js # WebSocket # 多线程 # 集群 # Hadoop # 大数据 # JDK # ElasticSearch # MySQL # 数据库 # Redis # Http # Nginx
SpringBoot中基于Redis的分布式锁实现(通过Redisson)
StreamSets安装(CentOS7)
  • 文章目录
  • 站点概览
Patrick

Patrick

不是在改BUG,就是在写BUG。

52 日志
9 分类
36 标签
RSS
E-mail
Creative Commons
© 2018 — 2023 Patrick
人生如逆旅|我亦是行人
鲁ICP备18043140号-1