Welcome to meizhangzheng's blog

大数据

2020-01-01

大数据

大数据
数据仓库定义

场景
- 新浪微博
- 微信
- 支付宝
。。。

工具
- hadoop
- hbase
- hive
- spark
。。。
Read All

云原生

2019-07-17

CNCF 云原生

云原生定义
服务网格定义
无服务器架构定义
- Serverless Architectures
云原生相关的项目、产品
参考资料

云原生定义

云原生技术有利于各组织在公有云，私有云和混合云等新型动态环境中，构建和运行可弹性扩展的应用。云原生的代表技术包括容器，服务网格，微服务，不可变基础设施和声明式 API。

这些技术能够构建容错性好，易于管理和便于观察的松耦合系统。结合可靠的自动化手段，云原生技术使工程师能够轻松地对系统作出频繁和可预测的重大变更。

云原生计算基金会（CNCF）致力于培育和维护一个厂商中立的开源生态系统，来推广云原生技术。我们通过将最前沿的模式民主化，让这些创新为大众所用。来源

服务网格定义

WHAT IS A SERVICE MESH?

    A service mesh is a dedicated infrastructure layer for handling service-to-service communication. It’s responsible for the reliable delivery of requests through the complex topology of services that comprise a modern, cloud native application. In practice, the service mesh is typically implemented as an array of lightweight network proxies that are deployed alongside application code, without the application needing to be aware. (But there are variations to this idea, as we’ll see.)

翻译: 来源

什么是服务网格？

    服务网格是用于处理服务到服务通信的专用基础设施层。它负责通过包含现代云本机应用程序的复杂服务拓扑来可靠地传递请求。实际上，服务网格通常实现为轻量级网络代理的数组，这些代理与应用程序代码一起部署，而不需要知道应用程序。（但我们会看到，这个想法有所不同。）

无服务器架构定义

Serverless Architectures

   Serverless architectures are application designs that incorporate third-party “Backend as a Service” (BaaS) services, and/or that include custom code run in managed, ephemeral containers on a “Functions as a Service” (FaaS) platform. By using these ideas, and related ones like single-page applications, such architectures remove much of the need for a traditional always-on server component. Serverless architectures may benefit from significantly reduced operational cost, complexity, and engineering lead time, at a cost of increased reliance on vendor dependencies and comparatively immature supporting services.

翻译：来源

   无服务器体系结构是包含第三方“后端即服务”（BaaS）服务的应用程序设计，和/或包括在“功能即服务”（FaaS）平台上的托管，临时容器中运行的自定义代码。通过使用这些想法以及相关的想法（如单页面应用程序），此类体系结构消除了对传统永远在线服务器组件的大量需求。无服务器架构可以从显着降低的运营成本，复杂性和工程提前期中受益，代价是增加了对供应商依赖性和相对不成熟的支持服务的依赖。

云原生相关的项目、产品

git地址：https://github.com/cncf/landscape.git

产品分类简介：

参考资料

Iaas、Paas、Saas 云服务的区别

Read All

K8s 安装

2018-11-01

Docker K8s 容器

Docker K8s 容器
- K8s基本
K8s基本

容器运用比较广泛，微服务、CI。

常见概念理解：容器（软件包）、Docker（容器管理平台）、K8s（容器集群管理、分布式容器管理平台）、Ingress

[1]
Read All

K8s 基本

2018-11-01

Docker K8s 容器

K8s基本
- 1. 组件
  - 1.0 k8s网络
    - 1.0.1 k8s网络通讯问题
    - 1.0.2 Pod

K8s基本

容器运用比较广泛，微服务、CI。

常见概念理解：容器（软件包）、Docker（容器管理平台）、K8s（容器集群管理、分布式容器管理平台）、Ingress

1. 组件

2021-08-24_K8s基础组件1

2021-08-24_K8s基础组件2 k8s 组件

1.0 k8s网络

1.0.1 k8s网络通讯问题

名称	通讯
容器间通信	即同一个Pod内多个容器间通信，通常使用loopback来实现。
Pod间通信	K8s要求,Pod和Pod之间通信必须使用Pod-IP直接访问另一个Pod-IP，它们关联在同一个Docker0网桥。
Pod与Service通信	即PodIP去访问ClusterIP，当然，clusterIP实际上是IPVS 或 iptables规则的虚拟IP，是没有TCP/IP协议栈支持的。但不影响Pod访问它
Service与集群外部 Client的通信	K8s中Pod提供的服务必须能被互联网上的用户所访问到

CNI(容器网络接口)

：K8s中提供的一种通用网络标准规范，因为k8s本身不提供网络解决方案。（kube-router、flannel、Calico、Docker原生的Overlay）

flannel：为Pod配置网络的(如:添加网络，配置网络，激活网络等守护进程。

1.0.2 Pod

//	K8s Pod的网络创建流程 
/*
    1.每个Pod除了创建时指定的容器外，都有一个kubelet启动时指定的基础容器
    2.kubelet创建基础容器，生成network namespace
    3.kubelet调用网络CNI driver，由它根据配置调用具体的CNI 插件(eg:calico,flannel)
    4.CNI插件给基础容器配置网络
    5.Pod中其他的容器共享使用基础容器的网络	
*/

Read All

Docker Jenkins

2018-10-03

Docker Jenkins 容器实战

Docker Jenkins
- 一、示例步骤
- 参考文档

Docker Jenkins

基于Docker，安装Jenkins。

建议直接安装（非Docker)

一、示例步骤

1. 拉去镜像(lts:发行稳定版本)


docker pull jenkins/jenkins:lts

2. 环境变量配置

#本文的挂载目录是home下
mkdir -p /usr/meizhangzheng/jenkins

3. 启动命令

#运用镜像启动容器命令

docker run -d -p 8080:8080 -p 50000:50000 -v /usr/meizhangzheng/jenkins:/var/jenkins_home --name jenkins --restart always --privileged=true  -u root jenkins/jenkins:lts

# 进入容器，查看密码
docker exec -it jenkins /bin/bash
cat /var/jenkins_home/secrets/initialAdminPassword

4. 完善镜像资源路径

# 修改hudson.model.UpdateCenter.xml 配置信息
sed -i  's/https:\/\/updates.jenkins.io\/update-center.json/https:\/\/mirrors.tuna.tsinghua.edu.cn\/jenkins\/updates\/update-center.json/g' /usr/meizhangzheng/jenkins/hudson.model.UpdateCenter.xml

# 修改updates/default.json 资源路径
sed -i  's/https:\/\/updates.jenkins.io\/download/https:\/\/mirrors.tuna.tsinghua.edu.cn\/jenkins/g' /usr/meizhangzheng/jenkins/updates/default.json

# 修改updates/default.json 资源路径
sed -i  's/http:\/\/www.google.com/http:\/\/www.baidu.com/g' /usr/meizhangzheng/jenkins/updates/default.json

手动修改密码

#项目路径
/usr/meizhangzheng/jenkins/users/
  
# 外层路径
/var/jenkins_home/users/  
  
#密码修改为：123456
<passwordHash>#jbcrypt:$2a$10$LxMm9HqAI/R4z7gL57qTouW/Mrz8uSaBpCGKvKc7K6dK.g/0yk/uq</passwordHash>

5. 插件

# 选择插件 安装 SSH 、Git、Maven ...

# 配置Github、Gitee 账户信息

# 选择自由项目

6. 事例分析

背景：构建前端项目失败;

2021-08-09_jenkins

分析：联系上下文分析，具体分析走到哪一步，是否项目代码或业务问题；据分析代码拉去没问题，是引用插件问题。

参考文档

Read All

Docker MySQL

2018-10-02

Docker MySQL 实战

Docker MySQL 容器实战

content

Docker MySQL

非 Docker 安装 MySQL

基于 Docker，安装 MySQL。

一、安装步骤

1.拉取镜像

# docker pull meizhangzheng/mysql:8.022
docker pull mysql:8.0.11

2. 初始化

2.1 配置文件路径

mkdir -p /usr/local/mysqlData/test/cnf
mkdir -p /usr/local/mysqlData/test/data
vi /usr/local/mysqlData/test/cnf/my.cnf

2.2 配置 my.cnf

配置来源与 mysql 8 的 my.ini 文件，修改了其中的路径信息

[client]

# pipe=
# socket=mysql=mysql=mysql=MYSQL

port=3306

[mysql]
no-beep=

# default-character-set=

# SERVER SECTION
# ----------------------------------------------------------------------
#
# The following options will be read by the MySQL Server. Make sure that
# you have installed the server correctly (see above) so it reads this
# file.=
#
# server_type=3
[mysqld]

# The next three options are mutually exclusive to SERVER_PORT below.
# skip-networking=
# enable-named-pipe=
# shared-memory=

# shared-memory-base-name=MYSQL

# The Pipe the MySQL Server will use
# socket=mysql=mysql=mysql=mysql=MYSQL

# The TCP/IP Port the MySQL Server will listen on
port=3306

# Path to installation directory. All paths are usually resolved relative to this.
# basedir="/usr/local/mysqlData/test/"

# Path to the database root 数据
datadir=/usr/local/mysqlData/test/data

# The default character set that will be used when a new schema or table is
# created and no character set is defined
# character-set-server=

# The default authentication plugin to be used when connecting to the server
default_authentication_plugin=caching_sha2_password

# The default storage engine that will be used when create new tables when
default-storage-engine=INNODB

# Set the SQL mode to strict
sql-mode="STRICT_TRANS_TABLES,NO_ENGINE_SUBSTITUTION"

# General and Slow logging.
log-output=FILE
general-log=0
general_log_file="test-mysql.log"
slow-query-log=1
slow_query_log_file="test-mysql-slow.log"
long_query_time=10

# Binary Logging.
# log-bin=

# Error Logging.
log-error="test-mysq.err"

# Server Id.
server-id=1

# Specifies the on how table names are stored in the metadata.
# If set to 0, will throw an error on case-insensitive operative systems
# If set to 1, table names are stored in lowercase on disk and comparisons are not case sensitive.
# If set to 2, table names are stored as given but compared in lowercase.
# This option also applies to database names and table aliases.
# NOTE: Modify this value after Server initialization won't take effect.
lower_case_table_names=1

# Secure File Priv.   注意该路径的配置
secure-file-priv=/var/lib/mysql

# The maximum amount of concurrent sessions the MySQL server will
# allow. One of these connections will be reserved for a user with
# SUPER privileges to allow the administrator to login even if the
# connection limit has been reached.
max_connections=151

# The number of open tables for all threads. Increasing this value
# increases the number of file descriptors that mysqld requires.
# Therefore you have to make sure to set the amount of open files
# allowed to at least 4096 in the variable "open-files-limit" in
# section [mysqld_safe]
table_open_cache=2000

# Maximum size for internal (in-memory) temporary tables. If a table
# grows larger than this value, it is automatically converted to disk
# based table This limitation is for a single table. There can be many
# of them.
tmp_table_size=16M

# How many threads we should keep in a cache for reuse. When a client
# disconnects, the client's threads are put in the cache if there aren't
# more than thread_cache_size threads from before.  This greatly reduces
# the amount of thread creations needed if you have a lot of new
# connections. (Normally this doesn't give a notable performance
# improvement if you have a good thread implementation.)
thread_cache_size=10

# *** MyISAM Specific options
# The maximum size of the temporary file MySQL is allowed to use while
# recreating the index (during REPAIR, ALTER TABLE or LOAD DATA INFILE.
# If the file-size would be bigger than this, the index will be created
# through the key cache (which is slower).
myisam_max_sort_file_size=100G

# If the temporary file used for fast index creation would be bigger
# than using the key cache by the amount specified here, then prefer the
# key cache method.  This is mainly used to force long character keys in
# large tables to use the slower key cache method to create the index.
myisam_sort_buffer_size=8M

# Size of the Key Buffer, used to cache index blocks for MyISAM tables.
# Do not set it larger than 30% of your available memory, as some memory
# is also required by the OS to cache rows. Even if you're not using
# MyISAM tables, you should still set it to 8-64M as it will also be
# used for internal temporary disk tables.
key_buffer_size=8M

# Size of the buffer used for doing full table scans of MyISAM tables.
# Allocated per thread, if a full scan is needed.
read_buffer_size=0

read_rnd_buffer_size=0

# *** INNODB Specific options ***
# innodb_data_home_dir=

# Use this option if you have a MySQL server with InnoDB support enabled
# but you do not plan to use it. This will save memory and disk space
# and speed up some things.
# skip-innodb=

# If set to 1, InnoDB will flush (fsync) the transaction logs to the
# disk at each commit, which offers full ACID behavior. If you are
# willing to compromise this safety, and you are running small
# transactions, you may set this to 0 or 2 to reduce disk I/O to the
# logs. Value 0 means that the log is only written to the log file and
# the log file flushed to disk approximately once per second. Value 2
# means the log is written to the log file at each commit, but the log
# file is only flushed to disk approximately once per second.
innodb_flush_log_at_trx_commit=1

# The size of the buffer InnoDB uses for buffering log data. As soon as
# it is full, InnoDB will have to flush it to disk. As it is flushed
# once per second anyway, it does not make sense to have it very large
# (even with long transactions).
innodb_log_buffer_size=1M

# InnoDB, unlike MyISAM, uses a buffer pool to cache both indexes and
# row data. The bigger you set this the less disk I/O is needed to
# access data in tables. On a dedicated database server you may set this
# parameter up to 80% of the machine physical memory size. Do not set it
# too large, though, because competition of the physical memory may
# cause paging in the operating system.  Note that on 32bit systems you
# might be limited to 2-3.5G of user level memory per process, so do not
# set it too high.
innodb_buffer_pool_size=8M

# Size of each log file in a log group. You should set the combined size
# of log files to about 25%-100% of your buffer pool size to avoid
# unneeded buffer pool flush activity on log file overwrite. However,
# note that a larger logfile size will increase the time needed for the
# recovery process.
innodb_log_file_size=48M

# Number of threads allowed inside the InnoDB kernel. The optimal value
# depends highly on the application, hardware as well as the OS
# scheduler properties. A too high value may lead to thread thrashing.
innodb_thread_concurrency=17

# The increment size (in MB) for extending the size of an auto-extend InnoDB system tablespace file when it becomes full.
innodb_autoextend_increment=64

# The number of regions that the InnoDB buffer pool is divided into.
# For systems with buffer pools in the multi-gigabyte range, dividing the buffer pool into separate instances can improve concurrency,
# by reducing contention as different threads read and write to cached pages.
innodb_buffer_pool_instances=8

# Determines the number of threads that can enter InnoDB concurrently.
innodb_concurrency_tickets=5000

# Specifies how long in milliseconds (ms) a block inserted into the old sublist must stay there after its first access before
# it can be moved to the new sublist.
innodb_old_blocks_time=1000

# It specifies the maximum number of .ibd files that MySQL can keep open at one time. The minimum value is 10.
innodb_open_files=300

# When this variable is enabled, InnoDB updates statistics during metadata statements.
innodb_stats_on_metadata=0

# When innodb_file_per_table is enabled (the default in 5.6.6 and higher), InnoDB stores the data and indexes for each newly created table
# in a separate .ibd file, rather than in the system tablespace.
innodb_file_per_table=1

# Use the following list of values: 0 for crc32, 1 for strict_crc32, 2 for innodb, 3 for strict_innodb, 4 for none, 5 for strict_none.
innodb_checksum_algorithm=0

# The number of outstanding connection requests MySQL can have.
# This option is useful when the main MySQL thread gets many connection requests in a very short time.
# It then takes some time (although very little) for the main thread to check the connection and start a new thread.
# The back_log value indicates how many requests can be stacked during this short time before MySQL momentarily
# stops answering new requests.
# You need to increase this only if you expect a large number of connections in a short period of time.
back_log=80

# If this is set to a nonzero value, all tables are closed every flush_time seconds to free up resources and
# synchronize unflushed data to disk.
# This option is best used only on systems with minimal resources.
flush_time=0

# The minimum size of the buffer that is used for plain index scans, range index scans, and joins that do not use
# indexes and thus perform full table scans.
join_buffer_size=256K

# The maximum size of one packet or any generated or intermediate string, or any parameter sent by the
# mysql_stmt_send_long_data() C API function.
max_allowed_packet=4M

# If more than this many successive connection requests from a host are interrupted without a successful connection,
# the server blocks that host from performing further connections.
max_connect_errors=100

# Changes the number of file descriptors available to mysqld.
# You should try increasing the value of this option if mysqld gives you the error "Too many open files".
open_files_limit=4161

# If you see many sort_merge_passes per second in SHOW GLOBAL STATUS output, you can consider increasing the
# sort_buffer_size value to speed up ORDER BY or GROUP BY operations that cannot be improved with query optimization
# or improved indexing.
sort_buffer_size=256K

# The number of table definitions (from .frm files) that can be stored in the definition cache.
# If you use a large number of tables, you can create a large table definition cache to speed up opening of tables.
# The table definition cache takes less space and does not use file descriptors, unlike the normal table cache.
# The minimum and default values are both 400.
table_definition_cache=1400

# Specify the maximum size of a row-based binary log event, in bytes.
# Rows are grouped into events smaller than this size if possible. The value should be a multiple of 256.
binlog_row_event_max_size=8K

# If the value of this variable is greater than 0, a replication slave synchronizes its master.info file to disk.
# (using fdatasync()) after every sync_master_info events.
sync_master_info=10000

# If the value of this variable is greater than 0, the MySQL server synchronizes its relay log to disk.
# (using fdatasync()) after every sync_relay_log writes to the relay log.
sync_relay_log=10000

# If the value of this variable is greater than 0, a replication slave synchronizes its relay-log.info file to disk.
# (using fdatasync()) after every sync_relay_log_info transactions.
sync_relay_log_info=10000

# Load mysql plugins at start."plugin_x ; plugin_y".
# plugin_load=

# The TCP/IP Port the MySQL Server X Protocol will listen on.
loose_mysqlx_port=33060

3.启动服务

# mysql 8 不推荐使用软链接

docker run --restart=always --privileged=true -itd -p 3307:3306 --name test_mysql -v /usr/local/mysqlData/test/conf/my.cnf:/etc/mysql/my.cnf -v /usr/local/mysqlData/test/data:/var/lib/mysql -e MYSQL_ROOT_PASSWORD=123456 mysql:8.0.11

# 参数说明：
# –restart=always： 当Docker 重启时，容器会自动启动。
# –privileged=true：容器内的root拥有真正root权限，否则容器内root只是外部普通用户权限
# -v /usr/local/mysqlData/test/log:/var/log/mysql 映射日志文件
# -v /usr/local/mysqlData/test/data:/var/lib/mysql 映射数据目录
# -v /usr/local/mysqlData/test/conf/my.cnf:/etc/mysql/my.cnf 映射配置文件

4. 配置远程连接

# 登录到test_mysql 容器
docker exec -it test_mysql bash

# 访问mysql (密码为初始密码)
myql -uroot -p

# 查询用户（可忽略）
select user,host,authentication_string from mysql.user;

# 设置权限（为root分配权限，以便可以远程连接）
update user set host = '%' where user = 'root';

grant all PRIVILEGES on *.* to root@'%' WITH GRANT OPTION;

# 由于Mysql5.6以上的版本修改了Password算法，这里需要更新密码算法，便于使用Navicat连接
grant all PRIVILEGES on *.* to root@'%' WITH GRANT OPTION;

# 更新密码算法（msyql 8.0 以上）
# ALTER user 'root'@'%' IDENTIFIED BY '123456' PASSWORD EXPIRE NEVER;
ALTER user 'root'@'%' IDENTIFIED WITH mysql_native_password BY '123456';

# 异常问题1：You must reset your password using ALTER USER statement before executing this statement

alter user 'root'@'localhost' identified by '123456'; #改密码方式一
alter user USER() identified by '123456'; 			   #改密码方式二

# 异常问题2： caching_sha2_password not be loaded:xxxx
 ALTER user 'root'@'localhost' IDENTIFIED WITH mysql_native_password BY '123456';

# 提交
FLUSH PRIVILEGES;

# 通过navicate 即可连接

二、基于安装包安装

# 安装发布的版本   archives(归档版本，无需再打包，解压、配置即可)
https://downloads.mysql.com/archives/community/

# 文件以 .tar.gz结尾

# 通过日志中查看mysql初始密码

select '自己有才是真的有';

参考文档

Docker 部署 Mysql8.0

Read All

7/24

Welcome to meizhangzheng's blog

场景

工具

云原生定义

服务网格定义

无服务器架构定义

Serverless Architectures

参考资料

K8s基本

K8s基本

1. 组件

1.0.1 k8s网络通讯问题

1.0.2 Pod

Docker Jenkins

一、示例步骤

1. 拉去镜像(lts:发行稳定版本)

2. 环境变量配置

3. 启动命令

4. 完善镜像资源路径

5. 插件

6. 事例分析

参考文档

Docker MySQL

一、安装步骤

1.拉取镜像

2. 初始化

2.1 配置文件路径

2.2 配置 my.cnf

3.启动服务

4. 配置远程连接

二、基于安装包安装

参考文档