找回Linux丢失的磁盘空间


最近碰到磁盘空间莫名丢失的问题

尝试找大的文件,做清理

find / -type f -size +100M

通过du命令检查根下的这个文件夹的占用大小,如果出现根目录下各个文件夹占用的空间不等于df命令查看到的磁盘空间占用率,需要考虑挂载点出现了非空挂载。

例如,挂载点/opt,原来有一个redis,redis占用10GB空间,如果新挂载一个设备到/opt,此时用du命令检查/opt目录,看不到redis占用的10GB空间,但是df -h,能够看到/节点有多余的10GB占用。

Advertisements

Typora配置正文、目录、侧边大纲中的标题自动编号


打开Typora偏好设置,打开主题文件夹,在主题文件夹中创建base.user.css文件,放置以下内容,则Typora自带的几个主题都会生效。

/**************************************
* Header Counters in TOC
**************************************/

/* No link underlines in TOC */
.md-toc-inner {
text-decoration: none;
}

.md-toc-content {
counter-reset: h1toc
}

.md-toc-h1 {
margin-left: 0;
font-size: 1.5rem;
counter-reset: h2toc
}

.md-toc-h2 {
font-size: 1.1rem;
margin-left: 2rem;
counter-reset: h3toc
}

.md-toc-h3 {
margin-left: 3rem;
font-size: .9rem;
counter-reset: h4toc
}

.md-toc-h4 {
margin-left: 4rem;
font-size: .85rem;
counter-reset: h5toc
}

.md-toc-h5 {
margin-left: 5rem;
font-size: .8rem;
counter-reset: h6toc
}

.md-toc-h6 {
margin-left: 6rem;
font-size: .75rem;
}

.md-toc-h1:before {
color: black;
counter-increment: h1toc;
content: counter(h1toc) “. “
}

.md-toc-h1 .md-toc-inner {
margin-left: 0;
}

.md-toc-h2:before {
color: black;
counter-increment: h2toc;
content: counter(h1toc) “. ” counter(h2toc) “. “
}

.md-toc-h2 .md-toc-inner {
margin-left: 0;
}

.md-toc-h3:before {
color: black;
counter-increment: h3toc;
content: counter(h1toc) “. ” counter(h2toc) “. ” counter(h3toc) “. “
}

.md-toc-h3 .md-toc-inner {
margin-left: 0;
}

.md-toc-h4:before {
color: black;
counter-increment: h4toc;
content: counter(h1toc) “. ” counter(h2toc) “. ” counter(h3toc) “. ” counter(h4toc) “. “
}

.md-toc-h4 .md-toc-inner {
margin-left: 0;
}

.md-toc-h5:before {
color: black;
counter-increment: h5toc;
content: counter(h1toc) “. ” counter(h2toc) “. ” counter(h3toc) “. ” counter(h4toc) “. ” counter(h5toc) “. “
}

.md-toc-h5 .md-toc-inner {
margin-left: 0;
}

.md-toc-h6:before {
color: black;
counter-increment: h6toc;
content: counter(h1toc) “. ” counter(h2toc) “. ” counter(h3toc) “. ” counter(h4toc) “. ” counter(h5toc) “. ” counter(h6toc) “. “
}

.md-toc-h6 .md-toc-inner {
margin-left: 0;
}

/**************************************
* Header Counters in Content
**************************************/

/** initialize css counter */
#write {
counter-reset: h1
}

h1 {
counter-reset: h2
}

h2 {
counter-reset: h3
}

h3 {
counter-reset: h4
}

h4 {
counter-reset: h5
}

h5 {
counter-reset: h6
}

/** put counter result into headings */
#write h1:before {
counter-increment: h1;
content: counter(h1) “. “
}

#write h2:before {
counter-increment: h2;
content: counter(h1) “.” counter(h2) “. “
}

#write h3:before, h3.md-focus.md-heading:before { /*override the default style for focused headings */
counter-increment: h3;
content: counter(h1) “.” counter(h2) “.” counter(h3) “. “
}

#write h4:before, h4.md-focus.md-heading:before {
counter-increment: h4;
content: counter(h1) “.” counter(h2) “.” counter(h3) “.” counter(h4) “. “
}

#write h5:before, h5.md-focus.md-heading:before {
counter-increment: h5;
content: counter(h1) “.” counter(h2) “.” counter(h3) “.” counter(h4) “.” counter(h5) “. “
}

#write h6:before, h6.md-focus.md-heading:before {
counter-increment: h6;
content: counter(h1) “.” counter(h2) “.” counter(h3) “.” counter(h4) “.” counter(h5) “.” counter(h6) “. “
}

/** override the default style for focused headings */
#write>h3.md-focus:before, #write>h4.md-focus:before, #write>h5.md-focus:before, #write>h6.md-focus:before, h3.md-focus:before, h4.md-focus:before, h5.md-focus:before, h6.md-focus:before {
color: inherit;
border: inherit;
border-radius: inherit;
position: inherit;
left: initial;
float: none;
top: initial;
font-size: inherit;
padding-left: inherit;
padding-right: inherit;
vertical-align: inherit;
font-weight: inherit;
line-height: inherit;
}

/**************************************
* Header Counters in sidebar
**************************************/
.sidebar-content {
counter-reset: h1
}

.outline-h1 {
counter-reset: h2
}

.outline-h2 {
counter-reset: h3
}

.outline-h3 {
counter-reset: h4
}

.outline-h4 {
counter-reset: h5
}

.outline-h5 {
counter-reset: h6
}

.outline-h1>.outline-item>.outline-label:before {
counter-increment: h1;
content: counter(h1) “. “
}

.outline-h2>.outline-item>.outline-label:before {
counter-increment: h2;
content: counter(h1) “.” counter(h2) “. “
}

.outline-h3>.outline-item>.outline-label:before {
counter-increment: h3;
content: counter(h1) “.” counter(h2) “.” counter(h3) “. “
}

.outline-h4>.outline-item>.outline-label:before {
counter-increment: h4;
content: counter(h1) “.” counter(h2) “.” counter(h3) “.” counter(h4) “. “
}

.outline-h5>.outline-item>.outline-label:before {
counter-increment: h5;
content: counter(h1) “.” counter(h2) “.” counter(h3) “.” counter(h4) “.” counter(h5) “. “
}

.outline-h6>.outline-item>.outline-label:before {
counter-increment: h6;
content: counter(h1) “.” counter(h2) “.” counter(h3) “.” counter(h4) “.” counter(h5) “.” counter(h6) “. “
}

以上参考:http://support.typora.io/Auto-Numbering/

spring-boot连接自制证书的exchange服务器发送邮件


一、pom管理依赖

<groupId>xxx</groupId>
<artifactId>xxx</artifactId>
<version>1.0-SNAPSHOT</version>

<parent>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-parent</artifactId>
<version>2.1.3.RELEASE</version>
</parent>

<dependencies>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-mail</artifactId>
</dependency>
</dependencies>

<build>
<plugins>
<plugin>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-maven-plugin</artifactId>
</plugin>
</plugins>
</build>

二、创建启动类

@SpringBootApplication
public class SpringBootMailApplication {

public static void main(String[] args) {
SpringApplication.run(SpringBootMailApplication.class, args);
}
}

三、创建邮件发送类

@Component
public class SenderMail {
private Logger logger = LogManager.getLogger(SenderMail.class);

@Autowired
private JavaMailSender javaMailSender;

private static String MAIL_SENDER = "xieshaohu@xxx.xxx";

@PostConstruct
public void sendHtmlMail() {
MimeMessage mimeMailMessage = null;
try {
mimeMailMessage = javaMailSender.createMimeMessage();
MimeMessageHelper mimeMessageHelper = new MimeMessageHelper(mimeMailMessage, true);
mimeMessageHelper.setFrom(MAIL_SENDER);
mimeMessageHelper.setTo("xxx@xxx.cn");
mimeMessageHelper.setSubject("测试邮件!");
StringBuilder sb = new StringBuilder();
sb.append("<h1>SpirngBoot测试邮件HTML</h1>")
.append("<p style='color:#F00'>你是真的太棒了!</p>")
.append("<p style='text-align:right'>右对齐</p>");
mimeMessageHelper.setText(sb.toString(), true);
javaMailSender.send(mimeMailMessage);
} catch (Exception e) {
logger.error("邮件发送失败", e);
}
}
}

四、编写application.yml配置文件

spring:
mail:
host: mail.xxx.com
port: 587 # exchange服务器时,必须设置此端口
username: xieshaohu
password: xxxxxx
default-encoding: UTF-8
jndi-name: mail/Session
properties:
mail:
smtp:
auth: true
starttls:
enable: false # 禁止TLS连接
ssl:
enable: false # 禁止ssl连接
socketFactory:
port: 587 # 设置端口
class: javax.net.ssl.SSLSocketFactory
fallback: true # 设置为true
debug: true
test-connection: false

五、配置文件特别说明:

  • 如果服务器支持不使用tls和ssl,则properties里的mail.smtp.starttls.enable就设置为false;mail.smtp.starttls.enable也设置为false
  • properties里的mail.smtp.socketFactory.fallback必须设置为true,否则会提示“javax.net.ssl.SSLException: Unrecognized SSL message, plaintext connection?”
  • 指定spring.mail.port为587,否则会提示:javax.mail.AuthenticationFailedException: 535 5.7.3 Authentication unsuccessful

六、最精简的连接成功的配置

spring:
mail:
host: mail.xxx.com
port: 587 # exchange服务器时,必须设置此端口
username: xieshaohu
password: xxxxxx
default-encoding: UTF-8
properties:
mail:
smtp:
auth: true
debug: true
test-connection: false

总结:

由于最初没有配置spring.mail.port,导致研究了一圈properties的配置,发送成功后,最后通过测试发现,设置好port端口为587,一切都可以正常执行了。

15分钟——在Docker启动Cloudera并开始体验


开始之前

  • 最新版Docker
  • 给Docker分配4GB内存
  • Docker可用的磁盘空间10GB

使用Docker可以快速体验CDH,并且依据localhost提供的案例,可以体验CDH在Sqoop,HDFS,Hive,Impala,Spark,Search方面的应用。安装预计耗时15分钟左右,体验上诉各种场景在2小时左右。

第一部分:安装CDH

1. 从Cloudera下载docker镜像

2. 解压缩之后导入

docker import cloudera-quickstart-vm-5.13.0-0-beta-docker.tar

3. 启动容器

docker run --hostname=quickstart.cloudera --privileged=true -t -i -d -p 8888:8888 -p 80:80 -p 7180:7180 -p 21050:21050 -p 50070:50070 -p 50075:50075 -p 50010:50010 -p 50020:50020 -p 8020:8020 [containerid] /usr/bin/docker-quickstart

第二部分:使用Sqoop从Mysql导入表,并使用impala进行SQL查询

1. 进入服务器

docker exec -t -i [container name] /bin/bash

2. 切换到cloudera用户

su - cloudera

3. 导入测试数据

[cloudera@quickstart ~]$ sqoop import-all-tables \
-m 1 \
--connect jdbc:mysql://quickstart:3306/retail_db \
--username=retail_dba \
--password=cloudera \
--compression-codec=snappy \
--as-parquetfile \
--warehouse-dir=/user/hive/warehouse \
--hive-import

4. 访问http://localhost:8888,打开hue,账号密码都是cloudera

5. 刷新impala缓存

invalidate metadata;

6. hue界面中执行SQL查询

其他应用场景可以参考http://localhost进行体验,后续还包含search应用,spark体验等。

说明:docker版本默认不启动clouder-manager,如果需要启动,将docker的内存分配到8GB,然后进入cdh的容器中,执行/home/cloudera/cloudera-manager –express,之后可以通过浏览器访问localhost:7180查看到Cloudera-manager

MacOS 设置终端和VIM颜色


终端颜色设置

终端颜色效果

新建~/.bash_profile文件,填充以下内容

export LS_OPTIONS=’–color=auto’
export CLICOLOR=’Yes’
export LSCOLORS=’cxfxcxdxbxegedabagGxGx’

LSCOLORS说明:
LSCOLORS中一共11组颜色设置,共22个设置,按照先后顺序,先前景色,后背景色,11组顺序如下:

  1. directory
  2. symbolic link
  3. socket
  4. pipe
  5. executable
  6. block special
  7. character special
  8. executable with setuid bit set
  9. executable with setgid bit set
  10. directory writable to others, with sticky bit
  11. directory writable to others, without sticky bit

LSCOLORS配置中,字母代表的颜色如下:

  • a 黑色
  • b 红色
  • c 绿色
  • d 棕色
  • e 蓝色
  • f 洋红色
  • g 青色
  • h 浅灰色
  • A 黑色粗体
  • B 红色粗体
  • C 绿色粗体
  • D 棕色粗体
  • E 蓝色粗体
  • F 洋红色粗体
  • G 青色粗体
  • H 浅灰色粗体
  • x 系统默认颜色

执行source ~/.bash_profile使配置生效

VIM颜色设置

vim开启颜色主题效果

复制默认配置到~/.vimrc

cp -r /usr/share/vim/vimrc ~/.vimrc

然后添加以下内容,其中syntax enableb必须要有,否则颜色不生效

syntax enable
colorscheme darkblue

其中darkblue是/usr/share/vim/vim80/colors/下的配置文件,可以选择一个自己喜欢的颜色主题

配置生效后,可以在vim窗口中执行:color或者:colorscheme查看当前的颜色主题

pagehelper-spring-boot-starter配置多数据源的分页


需求:同时连接Mysql和Oracle两个数据源,需要配置自动分页。

首先参考:https://www.cnblogs.com/xuwujing/p/8964927.html

补充说明,按照参考文档进行配置后,还是无法正确解析Oracle和Mysql的分页,所以,最终放弃通过application.yml配置文件进行配置,直接在Mysql和Oracle的SessionFactory分别写死如下代码:

Interceptor interceptor = new PageInterceptor();
Properties properties = new Properties(); properties.setProperty(“helperDialect”, “mysql”);//Oracle数据库时设置为oracle properties.setProperty(“reasonable”, “true”); interceptor.setProperties(properties);
sessionFactory.setPlugins(new Interceptor[] {interceptor});

最后SpringBoot启动时,设置排除自动配置:

@SpringBootApplication(exclude = PageHelperAutoConfiguration.class)

总结,两个数据源总的思路是:分别设置Mysql文件解析目录,设置不同的Mybatis配置,对应不同的sessionfactory和事务管理器,然后在sessionfactory中对pagehelper的方言进行正确设置,即可正在在两套数据源中分别进行切换。服务层调用Mapper的时候,是自动完成sessionFactory映射的,所以配置完后,在服务层调用的时候并不知晓数据的差异。

Oracle使用spool快速导出超大表


为了将Oracle的数据迁移到Mysql,使用Navicat类的工具迁移太慢,针对超1亿级别的个别表,使用spool先导出成CSV文件,然后在mysql中使用load命令装载CSV文件中的数据到表中。

注意点如下,一定要关掉命令输出,关掉之前,一秒钟1万条,关掉后,一秒钟大约20万条:

set SERVEROUT off
set TERM off

第二个注意点,要设置行的长度,防止换行,且需要设置截断行后多余的空白字符、防止分页

SET LINESIZE 2500
set trimspool on
set pagesize 0

outfile.sql文件命令如下:

set colsep ,
set feedback off
set heading off
set trimout on
set pagesize 0
set echo off
set SERVEROUT off
set TERM off
set trimspool on
SET LINESIZE 2500
spool /data/output.csv
select ‘”‘ || C1 || ‘”,”‘ || C2 || ‘”,”‘ || C3 || ‘”,”‘ || PKID || ‘”‘ from SCHEMA.TABLE_NAME;
spool off
exit

启动命令:

sqlplus -s user/password@orcl @outfile.sql

Mysql导入命令:

load data infile “/var/lib/mysql-files/output.csv” into table SCHEMA.TABLE_NAME fields terminated by ‘,’ enclosed by ‘”‘;

设置好行分割符和字段使用双引号括起来。