| 1 |
- {"name":"Canal","tagline":"阿里巴巴mysql数据库binlog的增量订阅&消费组件","body":"<div class=\"blog_content\">\r\n <div class=\"iteye-blog-content-contain\">\r\n<p style=\"font-size: 14px;\"> </p>\r\n<h1>背景</h1>\r\n<p style=\"font-size: 14px;\"> 早期，阿里巴巴B2B公司因为存在杭州和美国双机房部署，存在跨机房同步的业务需求。不过早期的数据库同步业务，主要是基于trigger的方式获取增量变更，不过从2010年开始，阿里系公司开始逐步的尝试基于数据库的日志解析，获取增量变更进行同步，由此衍生出了增量订阅&消费的业务，从此开启了一段新纪元。ps. 目前内部使用的同步，已经支持mysql5.x和oracle部分版本的日志解析</p>\r\n<p style=\"font-size: 14px;\"> </p>\r\n<p style=\"font-size: 14px;\">基于日志增量订阅&消费支持的业务：</p>\r\n<ol style=\"font-size: 14px;\">\r\n<li>数据库镜像</li>\r\n<li>数据库实时备份</li>\r\n<li>多级索引 (卖家和买家各自分库索引)</li>\r\n<li>search build</li>\r\n<li>业务cache刷新</li>\r\n<li>价格变化等重要业务消息</li>\r\n</ol>\r\n<h1>项目介绍</h1>\r\n<p style=\"font-size: 14px;\"> 名称：canal [kə'næl]</p>\r\n<p style=\"font-size: 14px;\"> 译意： 水道/管道/沟渠 </p>\r\n<p style=\"font-size: 14px;\"> 语言： 纯java开发</p>\r\n<p style=\"font-size: 14px;\"> 定位： 基于数据库增量日志解析，提供增量数据订阅&消费，目前主要支持了mysql</p>\r\n<p style=\"font-size: 14px;\"> </p>\r\n<h2>工作原理</h2>\r\n<h3 style=\"font-size: 14px;\">mysql主备复制实现</h3>\r\n<p><img src=\"http://dl.iteye.com/upload/attachment/0080/3086/468c1a14-e7ad-3290-9d3d-44ac501a7227.jpg\" alt=\"\"><br> 从上层来看，复制分成三步：</p>\r\n<ol>\r\n<li>master将改变记录到二进制日志(binary log)中（这些记录叫做二进制日志事件，binary log events，可以通过show binlog events进行查看）；</li>\r\n<li>slave将master的binary log events拷贝到它的中继日志(relay log)；</li>\r\n<li>slave重做中继日志中的事件，将改变反映它自己的数据。</li>\r\n</ol>\r\n<h3>canal的工作原理：</h3>\r\n<p><img width=\"590\" src=\"http://dl.iteye.com/upload/attachment/0080/3107/c87b67ba-394c-3086-9577-9db05be04c95.jpg\" alt=\"\" height=\"273\"></p>\r\n<p>原理相对比较简单：</p>\r\n<ol>\r\n<li>canal模拟mysql slave的交互协议，伪装自己为mysql slave，向mysql master发送dump协议</li>\r\n<li>mysql master收到dump请求，开始推送binary log给slave(也就是canal)</li>\r\n<li>canal解析binary log对象(原始为byte流)</li>\r\n</ol>\r\n<h1>架构</h1>\r\n<p><img width=\"548\" src=\"http://dl.iteye.com/upload/attachment/0080/3126/49550085-0cd2-32fa-86a6-f676db5b597b.jpg\" alt=\"\" height=\"238\" style=\"line-height: 1.5;\"></p>\r\n<p style=\"color: #333333; background-image: none; margin-top: 10px; margin-bottom: 10px; font-family: Arial, Helvetica, FreeSans, sans-serif;\">说明：</p>\r\n<ul style=\"line-height: 1.5; color: #333333; font-family: Arial, Helvetica, FreeSans, sans-serif;\">\r\n<li>server代表一个canal运行实例，对应于一个jvm</li>\r\n<li>instance对应于一个数据队列 （1个server对应1..n个instance)</li>\r\n</ul>\r\n<p>instance模块：</p>\r\n<ul style=\"line-height: 1.5; color: #333333; font-family: Arial, Helvetica, FreeSans, sans-serif;\">\r\n<li>eventParser (数据源接入，模拟slave协议和master进行交互，协议解析)</li>\r\n<li>eventSink (Parser和Store链接器，进行数据过滤，加工，分发的工作)</li>\r\n<li>eventStore (数据存储)</li>\r\n<li>metaManager (增量订阅&消费信息管理器)</li>\r\n</ul>\r\n<h3>数据对象格式：<a href=\"https://github.com/otter-projects/canal/blob/master/protocol/src/main/java/com/alibaba/otter/canal/protocol/EntryProtocol.proto\" style=\"font-size: 14px; line-height: 1.5; color: #bc2a4d; text-decoration: underline;\">EntryProtocol.proto</a>\r\n</h3>\r\n<pre name=\"code\" class=\"java\">Entry\r\n Header\r\n\t\tlogfileName [binlog文件名]\r\n\t\tlogfileOffset [binlog position]\r\n\t\texecuteTime [发生的变更]\r\n\t\tschemaName \r\n\t\ttableName\r\n\t\teventType [insert/update/delete类型]\r\n\tentryType \t[事务头BEGIN/事务尾END/数据ROWDATA]\r\n\tstoreValue \t[byte数据,可展开，对应的类型为RowChange]\r\n\t\r\nRowChange\r\n\tisDdl\t\t[是否是ddl变更操作，比如create table/drop table]\r\n\tsql\t\t[具体的ddl sql]\r\n\trowDatas\t[具体insert/update/delete的变更数据，可为多条，1个binlog event事件可对应多条变更，比如批处理]\r\n\t\tbeforeColumns [Column类型的数组]\r\n\t\tafterColumns [Column类型的数组]\r\n\t\t\r\nColumn \r\n\tindex\t\t\r\n\tsqlType\t\t[jdbc type]\r\n\tname\t\t[column name]\r\n\tisKey\t\t[是否为主键]\r\n\tupdated\t\t[是否发生过变更]\r\n\tisNull\t\t[值是否为null]\r\n\tvalue\t\t[具体的内容，注意为文本]</pre>\r\n<p>说明：</p>\r\n<ul>\r\n<li>可以提供数据库变更前和变更后的字段内容，针对binlog中没有的name,isKey等信息进行补全</li>\r\n<li>可以提供ddl的变更语句</li>\r\n</ul>\r\n<h1>QuickStart</h1>\r\n<h2>几点说明：(mysql初始化)</h2>\r\n<p>a. canal的原理是基于mysql binlog技术，所以这里一定需要开启mysql的binlog写入功能，并且配置binlog模式为row. </p>\r\n<pre class=\"java\" name=\"code\">[mysqld]\r\nlog-bin=mysql-bin #添加这一行就ok\r\nbinlog-format=ROW #选择row模式\r\nserver_id=1 #配置mysql replaction需要定义，不能和canal的slaveId重复</pre>\r\nb. canal的原理是模拟自己为mysql slave，所以这里一定需要做为mysql slave的相关权限.</div>\r\n<div class=\"iteye-blog-content-contain\">\r\n<pre class=\"java\" name=\"code\">CREATE USER canal IDENTIFIED BY 'canal'; \r\nGRANT SELECT, REPLICATION SLAVE, REPLICATION CLIENT ON *.* TO 'canal'@'%';\r\n-- GRANT ALL PRIVILEGES ON *.* TO 'canal'@'%' ;\r\nFLUSH PRIVILEGES;</pre>\r\n<p>针对已有的账户可通过grants查询权限：</p>\r\n<h2>启动步骤：</h2>\r\n<p>1. 下载canal</p>\r\n<p>下载部署包</p>\r\n<pre name=\"code\" class=\"java\">wget http://canal4mysql.googlecode.com/files/canal.deployer-1.0.0.tar.gz</pre>\r\n<p>or </p>\r\n<p>自己编译 </p>\r\n<pre name=\"code\" class=\"java\">git clone git@github.com:otter-projects/canal.git\r\ncd canal; \r\nmvn clean install -Dmaven.test.skip -Denv=release</pre>\r\n<p> 编译完成后，会在根目录下产生target/canal.deployer-$version.tar.gz </p>\r\n<p> </p>\r\n<p>2. 解压缩</p>\r\n<pre name=\"code\" class=\"java\">mkdir /tmp/canal\r\ntar zxvf canal.deployer-1.0.0.tar.gz -C /tmp/canal</pre>\r\n<p> </p>\r\n<p> 解压完成后，进入/tmp/canal目录，可以看到如下结构：</p>\r\n<p> </p>\r\n<pre name=\"code\" class=\"java\">drwxr-xr-x 2 jianghang jianghang 136 2013-02-05 21:51 bin\r\ndrwxr-xr-x 4 jianghang jianghang 160 2013-02-05 21:51 conf\r\ndrwxr-xr-x 2 jianghang jianghang 1.3K 2013-02-05 21:51 lib\r\ndrwxr-xr-x 2 jianghang jianghang 48 2013-02-05 21:29 logs</pre>\r\n<p> </p>\r\n<p>3. 配置修改</p>\r\n<p> </p>\r\n<p>公用参数： </p>\r\n<pre name=\"code\" class=\"shell\">vi conf/canal.properties</pre>\r\n<pre name=\"code\" class=\"java\">#################################################\r\n######### common argument ############# \r\n#################################################\r\ncanal.id= 1\r\ncanal.address=\r\ncanal.port= 11111\r\ncanal.zkServers=\r\n# flush data to zk\r\ncanal.zookeeper.flush.period = 1000\r\n## memory store RingBuffer size, should be Math.pow(2,n)\r\ncanal.instance.memory.buffer.size = 32768\r\n\r\n## detecing config\r\ncanal.instance.detecting.enable = false\r\ncanal.instance.detecting.sql = insert into retl.xdual values(1,now()) on duplicate key update x=now()\r\ncanal.instance.detecting.interval.time = 3 \r\ncanal.instance.detecting.retry.threshold = 3 \r\ncanal.instance.detecting.heartbeatHaEnable = false\r\n\r\n# support maximum transaction size, more than the size of the transaction will be cut into multiple transactions delivery\r\ncanal.instance.transactionn.size = 1024\r\n\r\n# network config\r\ncanal.instance.network.receiveBufferSize = 16384\r\ncanal.instance.network.sendBufferSize = 16384\r\ncanal.instance.network.soTimeout = 30\r\n\r\n#################################################\r\n######### destinations ############# \r\n#################################################\r\ncanal.destinations= example\r\n\r\ncanal.instance.global.mode = spring \r\ncanal.instance.global.lazy = true ##修改为false，代表立马启动\r\n#canal.instance.global.manager.address = 127.0.0.1:1099\r\ncanal.instance.global.spring.xml = classpath:spring/memory-instance.xml\r\n#canal.instance.global.spring.xml = classpath:spring/default-instance.xml</pre>\r\n<p> </p>\r\n<p>应用参数：</p>\r\n<pre name=\"code\" class=\"shell\">vi conf/example/instance.properties</pre>\r\n<pre name=\"code\" class=\"instance.properties\">#################################################\r\n## mysql serverId\r\ncanal.instance.mysql.slaveId = 1234\r\n\r\n# position info\r\ncanal.instance.master.address = 127.0.0.1:3306 #改成自己的数据库地址\r\ncanal.instance.master.journal.name = \r\ncanal.instance.master.position = \r\ncanal.instance.master.timestamp = \r\n\r\n#canal.instance.standby.address = \r\n#canal.instance.standby.journal.name =\r\n#canal.instance.standby.position = \r\n#canal.instance.standby.timestamp = \r\n\r\n# username/password\r\ncanal.instance.dbUsername = retl #改成自己的数据库信息\r\ncanal.instance.dbPassword = retl #改成自己的数据库信息\r\ncanal.instance.defaultDatabaseName = #改成自己的数据库信息\r\ncanal.instance.connectionCharsetNumber = 33 #改成自己的数据库信息\r\ncanal.instance.connectionCharset = UTF-8 #改成自己的数据库信息\r\n\r\n# table regex\r\ncanal.instance.filter.regex = .*\\\\..*\r\n\r\n#################################################\r\n</pre>\r\n<p> </p>\r\n<p> </p>\r\n<p> 说明：</p>\r\n<ul>\r\n<li>canal.instance.connectionCharset 代表数据库的编码方式对应到java中的编码类型，比如UTF-8，GBK , ISO-8859-1</li>\r\n<li>canal.instance.connectionCharsetNumber 代表数据库的编码方式对应mysql中的唯一id，详细的映射关系可查看：com.mysql.jdbc.CharsetMapping.INDEX_TO_CHARSET<br>针对常见的编码：<br>utf-8 <=> 33<br>gb2312 <=> 24<br>gbk <=> 28</li>\r\n</ul>\r\n<p>4. 准备启动</p>\r\n<p> </p>\r\n<pre name=\"code\" class=\"java\">sh bin/startup.sh</pre>\r\n<p> </p>\r\n<p>5. 查看日志</p>\r\n<pre name=\"code\" class=\"java\">vi logs/canal/canal.log</pre>\r\n<pre name=\"code\" class=\"java\">2013-02-05 22:45:27.967 [main] INFO com.alibaba.otter.canal.deployer.CanalLauncher - ## start the canal server.\r\n2013-02-05 22:45:28.113 [main] INFO com.alibaba.otter.canal.deployer.CanalController - ## start the canal server[10.1.29.120:11111]\r\n2013-02-05 22:45:28.210 [main] INFO com.alibaba.otter.canal.deployer.CanalLauncher - ## the canal server is running now ......</pre>\r\n<p> </p>\r\n<p> 具体instance的日志：</p>\r\n<pre name=\"code\" class=\"java\">vi logs/example/example.log</pre>\r\n<pre name=\"code\" class=\"java\">2013-02-05 22:50:45.636 [main] INFO c.a.o.c.i.spring.support.PropertyPlaceholderConfigurer - Loading properties file from class path resource [canal.properties]\r\n2013-02-05 22:50:45.641 [main] INFO c.a.o.c.i.spring.support.PropertyPlaceholderConfigurer - Loading properties file from class path resource [example/instance.properties]\r\n2013-02-05 22:50:45.803 [main] INFO c.a.otter.canal.instance.spring.CanalInstanceWithSpring - start CannalInstance for 1-example \r\n2013-02-05 22:50:45.810 [main] INFO c.a.otter.canal.instance.spring.CanalInstanceWithSpring - start successful....</pre>\r\n<p> </p>\r\n<p>6. 关闭</p>\r\n<pre name=\"code\" class=\"java\">sh bin/stop.sh</pre>\r\n<p> </p>\r\n<p>it's over. </p>\r\n</div>\r\n<h1>ClientExample</h1>\r\n<p>依赖配置：(目前暂未正式发布到mvn仓库，所以需要各位下载canal源码后手工执行下mvn clean install -Dmaven.test.skip)</p>\r\n<pre name=\"code\" class=\"java\"><dependency>\r\n <groupId>com.alibaba.otter</groupId>\r\n <artifactId>canal.client</artifactId>\r\n <version>1.0.0</version>\r\n</dependency></pre>\r\n<p> </p>\r\n<p>1. 创建mvn标准工程：</p>\r\n<pre name=\"code\" class=\"java\">mvn archetype:create -DgroupId=com.alibaba.otter -DartifactId=canal.sample</pre>\r\n<p> </p>\r\n<p>2. 修改pom.xml，添加依赖</p>\r\n<p> </p>\r\n<p>3. ClientSample代码</p>\r\n<pre name=\"code\" class=\"SimpleCanalClientExample\">package com.alibaba.otter.canal.sample;\r\n\r\nimport java.net.InetSocketAddress;\r\nimport java.util.List;\r\n\r\nimport com.alibaba.otter.canal.common.utils.AddressUtils;\r\nimport com.alibaba.otter.canal.protocol.Message;\r\nimport com.alibaba.otter.canal.protocol.CanalEntry.Column;\r\nimport com.alibaba.otter.canal.protocol.CanalEntry.Entry;\r\nimport com.alibaba.otter.canal.protocol.CanalEntry.EntryType;\r\nimport com.alibaba.otter.canal.protocol.CanalEntry.EventType;\r\nimport com.alibaba.otter.canal.protocol.CanalEntry.RowChange;\r\nimport com.alibaba.otter.canal.protocol.CanalEntry.RowData;\r\n\r\npublic class SimpleCanalClientExample {\r\n\r\n public static void main(String args[]) {\r\n // 创建链接\r\n CanalConnector connector = CanalConnectors.newSingleConnector(new InetSocketAddress(AddressUtils.getHostIp(),\r\n 11111), \"example\", \"\", \"\");\r\n int batchSize = 1000;\r\n int emptyCount = 0;\r\n try {\r\n connector.connect();\r\n connector.subscribe(\".*\\\\..*\");\r\n connector.rollback();\r\n int totalEmtryCount = 120;\r\n while (emptyCount < totalEmtryCount) {\r\n Message message = connector.getWithoutAck(batchSize); // 获取指定数量的数据\r\n long batchId = message.getId();\r\n int size = message.getEntries().size();\r\n if (batchId == -1 || size == 0) {\r\n emptyCount++;\r\n System.out.println(\"empty count : \" + emptyCount);\r\n try {\r\n Thread.sleep(1000);\r\n } catch (InterruptedException e) {\r\n }\r\n } else {\r\n emptyCount = 0;\r\n // System.out.printf(\"message[batchId=%s,size=%s] \\n\", batchId, size);\r\n printEntry(message.getEntries());\r\n }\r\n\r\n connector.ack(batchId); // 提交确认\r\n // connector.rollback(batchId); // 处理失败, 回滚数据\r\n }\r\n\r\n System.out.println(\"empty too many times, exit\");\r\n } finally {\r\n connector.disconnect();\r\n }\r\n }\r\n\r\n private static void printEntry(List<Entry> entrys) {\r\n for (Entry entry : entrys) {\r\n if (entry.getEntryType() == EntryType.TRANSACTIONBEGIN || entry.getEntryType() == EntryType.TRANSACTIONEND) {\r\n continue;\r\n }\r\n\r\n RowChange rowChage = null;\r\n try {\r\n rowChage = RowChange.parseFrom(entry.getStoreValue());\r\n } catch (Exception e) {\r\n throw new RuntimeException(\"ERROR ## parser of eromanga-event has an error , data:\" + entry.toString(),\r\n e);\r\n }\r\n\r\n EventType eventType = rowChage.getEventType();\r\n System.out.println(String.format(\"================> binlog[%s:%s] , name[%s,%s] , eventType : %s\",\r\n entry.getHeader().getLogfileName(), entry.getHeader().getLogfileOffset(),\r\n entry.getHeader().getSchemaName(), entry.getHeader().getTableName(),\r\n eventType));\r\n\r\n for (RowData rowData : rowChage.getRowDatasList()) {\r\n if (eventType == EventType.DELETE) {\r\n printColumn(rowData.getBeforeColumnsList());\r\n } else if (eventType == EventType.INSERT) {\r\n printColumn(rowData.getAfterColumnsList());\r\n } else {\r\n System.out.println(\"-------> before\");\r\n printColumn(rowData.getBeforeColumnsList());\r\n System.out.println(\"-------> after\");\r\n printColumn(rowData.getAfterColumnsList());\r\n }\r\n }\r\n }\r\n }\r\n\r\n private static void printColumn(List<Column> columns) {\r\n for (Column column : columns) {\r\n System.out.println(column.getName() + \" : \" + column.getValue() + \" update=\" + column.getUpdated());\r\n }\r\n }\r\n}</pre>\r\n<p> </p>\r\n<p>4. 运行Client</p>\r\n<p>首先启动Canal Server，可参加QuickStart : <a style=\"line-height: 1.5;\" href=\"/blogs/1796070\">http://agapple.iteye.com/blogs/1796070</a></p>\r\n<p>启动Canal Client后，可以从控制台从看到类似消息：</p>\r\n<pre name=\"code\" class=\"java\">empty count : 1\r\nempty count : 2\r\nempty count : 3\r\nempty count : 4</pre>\r\n<p> 此时代表当前数据库无变更数据</p>\r\n<p> </p>\r\n<p>5. 触发数据库变更</p>\r\n<pre name=\"code\" class=\"java\">mysql> use test;\r\nDatabase changed\r\nmysql> CREATE TABLE `xdual` (\r\n -> `ID` int(11) NOT NULL AUTO_INCREMENT,\r\n -> `X` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP,\r\n -> PRIMARY KEY (`ID`)\r\n -> ) ENGINE=InnoDB AUTO_INCREMENT=3 DEFAULT CHARSET=utf8 ;\r\nQuery OK, 0 rows affected (0.06 sec)\r\n\r\nmysql> insert into xdual(id,x) values(null,now());Query OK, 1 row affected (0.06 sec)</pre>\r\n<p> </p>\r\n<p>可以从控制台中看到：</p>\r\n<pre name=\"code\" class=\"java\">empty count : 1\r\nempty count : 2\r\nempty count : 3\r\nempty count : 4\r\n================> binlog[mysql-bin.001946:313661577] , name[test,xdual] , eventType : INSERT\r\nID : 4 update=true\r\nX : 2013-02-05 23:29:46 update=true</pre>\r\n<p> </p>\r\n<h2>最后：</h2>\r\n<p> 整个代码在附件中可以下载，如有问题可及时联系。 </p>\r\n</div>\r\n \r\n <div class=\"attachments\">\r\n<a href=\"http://dl.iteye.com/topics/download/7a893f19-bafb-313a-8a7a-e371a4265ad9\">canal.sample.tar.gz</a> (2.2 KB)\r\n </div>\r\n","google":"UA-10379866-5","note":"Don't delete this file! It's used internally to help with page regeneration."}
|
